Wednesday, June 2, 2010

Install AWStats on CentOS 5

SkyHi @ Wednesday, June 02, 2010

This page summarizes how to perform a basic installation of AWStats on CentOS 5. This tutorial assumes you are either logged in as root or have sudo.

Contents

[hide]

Install

Add RPMForge Yum Repository

Install & Configure Prerequisites

  • Install Apache
    • yum install httpd
  • Configure Apache to start on boot
    • /sbin/chkconfig --levels 345 httpd on
  • Configure iptables to allow Apache traffic
    • /sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT
    • /etc/init.d/iptables save
    • /etc/init.d/iptables restart

Install & Configure AWStats

  • Install AWStats
    • yum install awstats
  • Modify AWStats Apache Configuration
    • Edit /etc/httpd/conf.d/awstats.conf
Alias /awstats/icon/ /var/www/awstats/icon/

ScriptAlias /awstats/ /var/www/awstats/

DirectoryIndex awstats.pl
Options ExecCGI
order deny,allow
allow from all

  • Edit the following lines in the default awstats configuration file
    • /etc/awstats/awstats.localhost.localdomain.conf
SiteDomain="."

HostAliases=""
  • Rename config file
    • mv /etc/awstats/awstats.localhost.localdomain.conf /etc/awstats/awstats...conf
  • Update Statistics (Note: By default, statistics will be updated every hour.)
    • /usr/bin/awstats_updateall.pl now -confdir="/etc" -awstatsprog="/var/www/awstats/awstats.pl"
  • Start Apache
    • /etc/init.d/httpd start

Verify Install

  • Go to
    http://./awstats/awstats.pl?config=.






Stale NFS file handle

SkyHi @ Wednesday, June 02, 2010
Same problem here, check this out... I got this after a system crash on debian lenny; after which i've reinstalled the root partition but kept /home (which is on a different partition).

fsck.ext3 says that /home is clean...







igi:~$ ls -l

ls: cannot access bin: Stale NFS file handle

ls: cannot access PICTURES_nortel_tmp: Stale NFS file handle

total 78568

d????????? ? ? ? ? ? bin

d????????? ? ? ? ? ? PICTURES_nortel_tmp

drwxr-xr-x 2 penguin penguin 4096 2008-11-22 23:06 TESTATICA

-rw-r--r-- 1 penguin penguin 31395624 2008-12-03 16:25 test.avi

-rw-r--r-- 1 penguin penguin 14698436 2008-12-04 15:36 test_divx_mp3.avi

-rw-r----- 1 penguin penguin 4240419 2008-11-19 22:43 toolsdev.tar.bz2

-rw-r--r-- 1 penguin penguin 5366 2008-10-19 13:29 tourpart7.html

drwxr-xr-x 10 penguin penguin 12288 2008-12-04 23:25 UNIX_HELP






Notice the weired directories with: d?????????

These dirs give me the: "Stale NFS file handle" error



igi:~$ rm -rf PICTURES_nortel_tmp

rm: cannot remove `PICTURES_nortel_tmp': Stale NFS file handle






Can't remove them or "ls -l" them or anything else (neither as root or when booted from a live distro, etc...)



Any insight on solving this issue (other than salvaging whats left on the /home partition and then recreating a new partition and or the file system) would be appreciated


Solved it:



Even though the first time i did fsck.ext3 on the home partition.... it
said it was clean but then when i tried with "force"... it found a bunch
of "problems". after fsck finished (and i had to press "y" <yes>
a few hundred times... it got rid (deleted the weired directories with:
"Stale NFS file handle")



umount /dev/hde

fsck.ext -f /dev/hde

REFERENCES
http://www.linuxquestions.org/questions/linux-software-2/cant-get-rid-of-stale-nfs-file-handle-684475/

Install Awstats on Ubuntu

SkyHi @ Wednesday, June 02, 2010

This has been tested with Ubuntu server 8.04 (Hardy Heron).

Installation

sudo aptitude install awstats

To see the country of your visitors (not required)

sudo aptitude install libnet-ip-perl
sudo aptitude install libgeo-ipfree-perl

Configuration

Let's assume you want to analyze the Apache 2 log file of your website “toto.com” (It can be just an IP address).

Create configuration file

sudo cp /etc/awstats/awstats.conf /etc/awstats/awstats.toto.com.conf

Update these parameters

# apache2
LogFile="/var/log/apache2/access.log"

# domain name
SiteDomain="toto.com"
HostAliases="localhost 127.0.0.1 toto.com"

LogFormat=1

You can also activate these plugins

LoadPlugin="tooltips
LoadPlugin="geoipfree"

Run

sudo /usr/lib/cgi-bin/awstats.pl -config=toto.com -update

You should get something like this

Create/Update database for config "/etc/awstats/awstats.toto.com.conf" by AWStats version 6.7 (build 1.892)
From data in log file "/var/log/apache2/access.log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 191338
Found 0 dropped records,
Found 24 corrupted records,
Found 0 old records,
Found 191314 new qualified records.

Configure Apache to view the statistics

Create /etc/apache2/conf.d/statistics

Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
ScriptAlias /statistics/ /usr/lib/cgi-bin/
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

Restart Apache

sudo /etc/init.d/apache2 restart

View your statistics

or (in case you have more than one config file)

Update your statistics with cron

In /etc/crontab add

# update every 10 min
*/10 * * * * root /usr/lib/cgi-bin/awstats.pl -config=toto.com -update > /dev/null

How to reset awstats

Just delete the files in /var/lib/awtstats (directory defined by DirData parameter)

sudo su - root
cd /var/lib/awstats
rm *
logout

Notes

  • you have to be root because for some reason, sudo rm -R awstats/* wouldn't work.
  • if you get the message Never updated (See 'Build/Update' on awstats_setup.html page) whereas the awstats files are there in /var/lib/awtstats, it's probably a permission problem on this folder.

Source: http://awstats.sourceforge.net/docs/awstats_faq.html#RESET

Troubleshooting

awstats "Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats. "

Try to this URL (with your values): http://toto.com/statistics/awstats.pl?config=toto.com




References



Install AWStats on CentOS 5.3

SkyHi @ Wednesday, June 02, 2010

From AWstats website: “AWStats is a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar, IIS (W3C log format) and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers.”


The installation procedure is done on a typical LAMP install on CentOS 5.3 x64_86

Your apache needs the combined log directive for the virtual host you are going to monitor with AWStats, so in your virtualhost you must have something like:

CustomLog logs/example.com-access_log combined

If you use the common log format some statistic features (browsers, os, keywords…) can’t work.


In this tutorial I am using example.com so change it with your own domain and docroot location.


As root install the rpmforge repository:


# wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

# rpm -Uvh rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm


# yum install yum-priorities # if you have other repos you might want to add the priority options in the .repo file so you don’t mess up packages. More info


In case you are using i386 change the .x86_64 with .i386


# yum install awstats


Optional: (To see what countries are visiting your website)


# yum install perl-Geo-IP

# yum install GeoIP-data


Prepare the configuration file:


# cd /etc/awstats

# mv awstats.model.conf awstats.model.bak # Keep a backup of the original

# mv awstats.localhost.localdomain.conf awstats.example.com.conf # Change it to your domain


Edit the configuration file:


# vi /etc/awstats/awstats.example.com.conf


In line 51: change LogFile=”/var/log/httpd/access_log” to the location of the access_log file you want to gather statistics.


Ex: LogFile=”/var/log/httpd/example.com-access_log”


In line 153: change SiteDomain=”localhost.localdomain” to SiteDomain=”example.com


In line 168: change HostAliases=”localhost 127.0.0.1″ to HostAliases=”www.example.com”


In line 203: change DirData=”/var/www/awstats” to your docroot.

Ex: DirData=”/var/www/htdocs/example.com/awstats”


If you installed the Geo-IP plugins you need to enable it:


In line 1305 change #LoadPlugin=”geoip GEOIP_STANDARD /pathto/GeoIP.dat” to LoadPlugin=”geoip GEOIP_STANDARD /var/lib/GeoIP/GeoIP.dat”


Move the awstats directory to your docroot:


//uncessary, check http://shebangme.blogspot.com/2010/06/install-awstats-on-centos-5.html

# mv /var/www/awstats /var/www/htdocs/example.com


Change the permissions to the awstats directory:


# chown -R localuser:apache /var/www/htdocs/example.com/awstats


Edit the awstats.conf to setup the aliases and restrictions:


# vi /etc/httpd/conf.d/awstats.conf


Change the Alias, ScriptAlias and Directory location to where the awstats is in your docroot.


ex:


Alias /awstats/icon/ /var/www/htdocs/example.com/awstats/icon/

ScriptAlias /awstats/ /var/www/htdocs/example.com/awstats/



<Directory /var/www/htdocs/example.com/awstats/>

DirectoryIndex awstats.pl

Options ExecCGI

order deny,allow

deny from all

allow from 192.168.1. 66.80.199.111 # Allow access to awstate from the 192.168.1.0/24 subnet only

</Directory>


#Alias /css/ /var/www/awstats/css/

#Alias /js/ /var/www/awstats/js/


Reload Apache:


# service httpd reload


In case you don’t have the Options FollowSymLinks in httpd.conf, awstats will not work so you need to add it in the /etc/httpd/conf.d/awstats.conf above.

……..

Options ExecCGI

Options FollowSymLinks

……..


Update your statistics:


# cd /var/www/htdocs/example.com/awstats


# perl awstats.pl -config=example.com -update


If you see something like:


Create/Update database for config “/etc/awstats/awstats.example.com.conf” by AWStats version 6.9 (build 1.925)

From data in log file “/var/log/httpd/example.com-access_log”…

Phase 1 : First bypass old records, searching new record…

Direct access after last parsed record (after line 4)

Jumped lines in file: 4

Found 4 already parsed records.

Parsed lines in file: 13

Found 0 dropped records,

Found 0 corrupted records,

Found 0 old records,

Found 13 new qualified records.


Then everything went fine.


If you changed your logs from common to combined it might throw some errors when you update awstats so you need to clean the log file to have only the combined logs:


# cat /dev/null > /var/log/httpd/example.com-access_log


run the update again:


# perl awstats.pl -config=example.com -update

Update for config "/etc/awstats/awstats.example.com.conf"

With data in log file ""

Phase 1:

Seaching new records from beginning of log file

Phase 2:

Found 2389 new qualified records.


Navigate to http://example.com/awstats/awstats.pl


If the update time on the upper left hand corner is the same as the time you run the awstats.pl update and you can see the statistics, then everything works fine.


Setup the cron job (it will run hourly and redirect the output to /dev/null so it’s not going to send an email to root every hour) :


cd /etc/cron.hourly

echo “perl /var/www/htdocs/example.com/awstats/awstats.pl -config=example.com -update > /dev/null 2>&1″ > awstats.sh

chmod 755 awstats.sh


test the script:


./awstats.sh

Refresh your browser and if the update time changes then you are all set.


Enjoy the statistics on your website!


REFERENCES

http://beginlinux.com/blog/2009/04/awstats-graphical-view-of-your-logs/

Parse log file with most accessed ip

SkyHi @ Wednesday, June 02, 2010

1. grep "30/May" access_log > access_log.1
2. cut -d' ' -f1 access_log.1 > access_log.2
3. sort access_log.2|uniq -c|sort -n


Live:
netstat -ntu |awk '{print $5}'| cut -d: -f1 | sort | uniq -c |sort -nr

Find the most accessed ip
awk '{print $1}' 10stillhack.txt |cut -d: -f1 | sort | uniq -c |sort -nr|more

def CalculateApacheIpHits(logfile_pathname):
    IpHitListing = {}
    Contents = open(logfile_pathname, "r").xreadlines( )
    for line in Contents:
        Ip = line.split(" ")[0]
        if 6 < len(Ip) <= 15:
            IpHitListing[Ip] = IpHitListing.get(Ip, 0) + 1
    return IpHitListing

def TimeSpan(logfile_pathname):
    Dates = open(logfile_pathname, "r").readline( )
    Last  = open(logfile_pathname, "r").readlines( )
    firstdate = Dates.split(" ")[3]
    lastdate = Last[len(Last)-1].split(" ")[3]
    print ""
    print "Log covers the dates of: " + firstdate[1:] + " - " + lastdate[1:]
    print ""

TimeSpan("access.log.19")

HitsDictionary = CalculateApacheIpHits("access.log.19")
width = 10
ip = "IP Address"
hits = "Hits"
print '%15s         %5s' % (ip, hits)
print '     ----------          ----'
for key in HitsDictionary.keys():
    print '%15s    ->   %5s' % (str(key), str(HitsDictionary[key]))


Kill all process:
#ps ax -o user,pid |grep 'postfix' |awk '{print $2}' |xargs -l
#ps ax -o user,pid |grep 'postfix' |awk '{print $2}' |xargs -r kill -9
#killall -9 PROCESS

Apache process memory usage
ps -ylC httpd --sort:rss

sort the top entries in your access.log
awk '{print $1}' /var/log/http/access_log | sort |uniq -c |sort -n
netstat -ta |grep ESTABLISHED
sar -w
sar -I SUM
sar -d 5 0
REFERENCES
http://linuxgazette.net/123/vishnu.html
http://www.devside.net/articles/apache-performance-tuning
http://serverfault.com/questions/231940/high-linux-loads-on-low-cpu-memory-usage

Web Technologies

SkyHi @ Wednesday, June 02, 2010

HOWTO: Basic Apache performance tuning (httpd)

SkyHi @ Wednesday, June 02, 2010
This article will provide some basic apache (httpd) performance tuning. The objective of this article is to perform simple edits that will allow your (dv) server to run with less memory and to ensure higher availability under heavier loads.

This is Part 1 of a series of articles on how to tune your (dv) for better performance. Please see (dv) HOWTO: Performance tuning (Optimization) for additional articles in this series.

Symptoms:

  • Frequent apache (httpd) crashes
  • Slow server response
  • "Server unreachable" errors
  • kmemsize warnings in your QoS Alerts (Plesk Control Panel under Virtuozzo)
  • privvmpages warnings in your QoS Alerts (Plesk Control Panel under Virtuozzo)

LINKS:

http://httpd.apache.org/docs/2.2/ Apache Doumentation site
http://httpd.apache.org/docs/2.2/mod/core.html#timeout Documentation on the Timeout setting.
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients Documentation on MaxClients


DETAILS:

Apache Timeout setting

The timeout setting is the number of seconds before data "sends" or "receives" (to or from the client) time out. Having this set to a high number forces site visitors to "wait in line" which adds extra load to the server.
  1. First backup the file with the following command:

    cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.bkup.kb-246.1
  2. The following command opens httpd.conf and searches for the term Timeout:

    vi +/Timeout /etc/httpd/conf/httpd.conf 
    You should see the following lines:

    # Timeout: The number of seconds before receives and sends time out.
    Timeout 120
  3. Edit the section that reads:

    Timeout 120
    to a more reasonable value like the following:

    Timeout 20
  4. Save this file by pressing the escape key followed by :wq! which will save the file and exit vi.
  5. Restart Apache to apply the changes:

    /etc/init.d/httpd restart

Max Client Settings

On (dv) Dedicated-Virtual v2.0 Servers:

  1. First backup the file with the following command:

    cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.bkup.kb-246.2
  2. The following command opens httpd.conf and searches for MaxClients:

    vi +/MaxClients /etc/httpd/conf/httpd.conf
    You should see the following section marked prefork MPM.
  3. Edit the section that reads:


    StartServers 2
    MinSpareServers 1
    MaxSpareServers 5
    MaxClients 10
    MaxRequestsPerChild 1000


    to something more like this:
    • For Base servers:


      StartServers 1
      MinSpareServers 1
      MaxSpareServers 3
      MaxClients 50
      MaxRequestsPerChild 1000

    • For Rage servers:


      StartServers 2
      MinSpareServers 2
      MaxSpareServers 5
      MaxClients 100
      MaxRequestsPerChild 1000

    • For Extreme servers:


      StartServers 2
      MinSpareServers 2
      MaxSpareServers 5
      MaxClients 200
      MaxRequestsPerChild 1000

  4. Save this file and restart Apache to apply the changes:

    /etc/init.d/httpd restart

On (dv) Dedicated-Virtual v3.x Servers:

  1. First backup the file with the following command:


    cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.bkup.kb-246.2
  2. The following command opens httpd.conf and searches for MaxClients:


    vi +/MaxClients /etc/httpd/conf/httpd.conf
    You should see the following section marked prefork MPM.
  3. Edit the section that reads:


    StartServers 1
    MinSpareServers 1
    MaxSpareServers 20
    ServerLimit 20
    MaxClients 150
    MaxRequestsPerChild 4000

    to something more like this:
    • For Base servers:


      StartServers 1
      MinSpareServers 1
      MaxSpareServers 3
      ServerLimit 50
      MaxClients 50
      MaxRequestsPerChild 4000

    • For Rage servers:


      StartServers 2
      MinSpareServers 2
      MaxSpareServers 5
      ServerLimit 100
      MaxClients 100
      MaxRequestsPerChild 4000

    • For Extreme servers:


      StartServers 2
      MinSpareServers 2
      MaxSpareServers 5
      ServerLimit 200
      MaxClients 200
      MaxRequestsPerChild 4000


    • For Nitro servers:


      StartServers 20
      MinSpareServers 20
      MaxSpareServers 50
      ServerLimit 500
      MaxClients 500
      MaxRequestsPerChild 4000

  4. Save this file and restart Apache to apply the changes:

    /etc/init.d/httpd restart

NOTE:

If you are running into frequent Apache crashes you can check the error_log for MaxClients-related problems.

To check your error logs to see if you have MaxClient issues you can run the following command as root:

grep -i maxclient /var/log/httpd/error_log*
If this command returns any results after making the above changes you may need to fine-tune the MaxClients variables further.

The official documentation on these settings and many others can be found here:

http://httpd.apache.org/docs-2.0/mod/mpm_common.html

Revisions:

07-20-2009: Minor Fixes