Wednesday, June 2, 2010

Parse log file with most accessed ip

SkyHi @ Wednesday, June 02, 2010

1. grep "30/May" access_log > access_log.1
2. cut -d' ' -f1 access_log.1 > access_log.2
3. sort access_log.2|uniq -c|sort -n


Live:
netstat -ntu |awk '{print $5}'| cut -d: -f1 | sort | uniq -c |sort -nr

Find the most accessed ip
awk '{print $1}' 10stillhack.txt |cut -d: -f1 | sort | uniq -c |sort -nr|more

def CalculateApacheIpHits(logfile_pathname):
    IpHitListing = {}
    Contents = open(logfile_pathname, "r").xreadlines( )
    for line in Contents:
        Ip = line.split(" ")[0]
        if 6 < len(Ip) <= 15:
            IpHitListing[Ip] = IpHitListing.get(Ip, 0) + 1
    return IpHitListing

def TimeSpan(logfile_pathname):
    Dates = open(logfile_pathname, "r").readline( )
    Last  = open(logfile_pathname, "r").readlines( )
    firstdate = Dates.split(" ")[3]
    lastdate = Last[len(Last)-1].split(" ")[3]
    print ""
    print "Log covers the dates of: " + firstdate[1:] + " - " + lastdate[1:]
    print ""

TimeSpan("access.log.19")

HitsDictionary = CalculateApacheIpHits("access.log.19")
width = 10
ip = "IP Address"
hits = "Hits"
print '%15s         %5s' % (ip, hits)
print '     ----------          ----'
for key in HitsDictionary.keys():
    print '%15s    ->   %5s' % (str(key), str(HitsDictionary[key]))


Kill all process:
#ps ax -o user,pid |grep 'postfix' |awk '{print $2}' |xargs -l
#ps ax -o user,pid |grep 'postfix' |awk '{print $2}' |xargs -r kill -9
#killall -9 PROCESS

Apache process memory usage
ps -ylC httpd --sort:rss

sort the top entries in your access.log
awk '{print $1}' /var/log/http/access_log | sort |uniq -c |sort -n
netstat -ta |grep ESTABLISHED
sar -w
sar -I SUM
sar -d 5 0
REFERENCES
http://linuxgazette.net/123/vishnu.html
http://www.devside.net/articles/apache-performance-tuning
http://serverfault.com/questions/231940/high-linux-loads-on-low-cpu-memory-usage