Search Site:

About

Linux

Printers?

Programming

Windows?

Download

Skins

Edit - To Do - AllRecentChanges

Recent Changes Printable View Page History Edit Page

This page is part of the EmailServer article.

Maillog analysis with AWStats

Maillogs are mighty important when troubleshooting or to check if your server is properly configured for its load.
The problem with raw maillog is that it quickly becomes hard to get any time of big picture view of what's going on: you can easily trace individual connections, but you won't know how many emails you send every day, how many megabytes are being moved or how many DNS errors your system is having.

AWStats is a generic log analysis package that offers a nice web interface and can be made to analyse almost any type of logs. It's easy to extend too and quite simple to install.

I don;t recommend using the RPM from the AWStat website has it has been packaged differently for Fedora and if you use automatic update, your config will change. Just use yum to get it and make a virgin configuration file:

# yum install awstats

#  echo alias awstats_updateall.pl=\
"'awstats_updateall.pl -awstatsprog=/var/www/awstats/awstats.pl'" \
>> ~/.bashrc

# cd /etc/awstats/
# cp awstats.model.conf awstats.mail.conf

The second line creates an alias for the awstats_updateall.pl to include the path to the awstats.pl script. This is necessary as the Fedora version of AWStats moved the files to other locations but some scripts still expect them to be at the default one.

Then edit the /etc/awstats/awstats.mail.conf file (don't forget to change the mail.example.com to your server hostname):

LogFile="perl /usr/bin/maillogconvert.pl standard < /var/log/maillog |"
LogType=M
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
SiteDomain="mail.example.com"
LevelForBrowsersDetection=0
LevelForOSDetection=0
LevelForRefererAnalyze=0
LevelForRobotsDetection=0
LevelForWormsDetection=0
LevelForSearchEnginesDetection=0
LevelForFileTypesDetection=0
ShowMenu=1
ShowSummary=HB
ShowMonthStats=HB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=0
ShowHostsStats=HBL
ShowAuthenticatedUsers=0
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowSessionsStats=0
ShowPagesStats=0
ShowFileTypesStats=0
ShowFileSizesStats=0
ShowBrowsersStats=0
ShowOSStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1

Now make AWStats update its database regularly from a cron job (crontab -e), for instance, every 2h for mail analysis and every 3h for the web logs analysis:

00 */2 * * * /var/www/awstats/awstats.pl -update -config=mail
00 */3 * * * /var/www/awstats/awstats.pl -update -config=localhost.localdomain

Then we can construct the database right now so we can access it for the remaining of our configuration:

#  awstats_updateall.pl now

If there are errors, double-check that you entered all the correct options in the configuration file above. If the script says it can't find awstats.pl, make sure your ~/.bashrc file contains the alias we defined above (there should be no spaces on either side or the = sign) and that you have logged off and back in to ensure the aliases were read (the ~/.bashrc file is read each time you login).

AWStats Integration in Apache

So now we have AWStats installed and updating its database regularly but what we still need is to display the stats in a convenient way.

The default installation should have added the following lines to your /etc/httpd/conf.d/awstats.conf file. to make this set up a bit more secure, we're going to allow access only from web browsers on our LAN and maybe from our other office at 212.87.250.3, all others will get an Access Denied page instead:

Alias /awstats/icon/ /var/www/awstats/icon/
ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
    DirectoryIndex awstats.pl
    Options ExecCGI
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
</Directory>

Now, to access AWStats, you need to point your web browser to http://mail.example.com/awstats/awstats.pl?config=mail .
However, if you configured SquirrelMail in a similar way as we did, this will probably not work as we have been rewritting our web requests to ensure people were using HTTPS.

Our solution, amongst many possible, was to add a subdomain mailstats.example.com pointing to our mail server and add the following rewrite rules in the /etc/httpd/conf.d/awstats.conf :

NameVirtualHost *:80
<Virtualhost *:80>
    ServerName mailstats.faiveley-fareast.com
    DocumentRoot "/var/www/awstats"
    Alias /awstats/icon/ /var/www/awstats/icon/
    ScriptAlias /awstats/ /var/www/awstats/
    <Directory /var/www/awstats>
        DirectoryIndex awstats.pl
        Options ExecCGI
        order deny,allow
        deny from all
        Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
    </Directory>

    RewriteEngine   on
    RewriteLog      "/var/log/httpd/rewrite_log"
    RewriteLogLevel 4

    RewriteCond %{REQUEST_URI} ^/mail/?$|^/?$ [NC]
    RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
    RewriteRule (^.*$)   http://%1/awstats/awstats.pl?config=mail

    RewriteCond %{REQUEST_URI} ^/web/?$ [NC]
    RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
    RewriteRule (^.*$)   http://%1/awstats/awstats.pl?config=localhost.localdomain

    RewriteCond %{HTTPS} off
    RewriteCond %{HTTP_HOST} ^(mail\..*)$ [NC]
    RewriteRule (^.*$)   https://%1 [L,R]

    RewriteCond %{HTTP_HOST} ^webmail\.(.*)$ [NC]
    RewriteRule (^.*$)   https://mail.%1 [L,R]
</Virtualhost>

Requests to the web server wil be transformed as such :

  • http://mailstats.example.com or http://mailstats.example.com/mail
    -> http://mailstats.example.com/awstats/awstats.pl?config=mail
  • http://mailstats.example.com/web
    -> http://mailstats.example.com/awstats/awstats.pl?config=localhost.localdomain

Now, providing we created the mailstats sub-domain, we can access both our mail and web statistics without having to remember those long URLs.

Extending AWStats

AWStats by default will provide quite a bit of useful information, but it will also ignore some that it won't understand, like how much spam or viruses were found.

How to integrate this information into the stats collected by AWStats will be developped in the near future.

Resources

< TroubleShooting | EmailServer

Comments
RulfoMonday 05 June 2006, at 02:22 GMT+8 [X]
Great article, i hope advanced articule could be released soon. Bye
CubesSaturday 14 August 2010, at 20:43 GMT+8 [X]
Thanks just what I needed.
Enter your comment (no links allowed): Author:

Edit Page - Page History - Printable View - Recent Changes - WikiHelp - Search - RSS -
Page last modified on Friday 29 July 2005, at 02:32 GMT+8 - Viewed 3942 times