AW Stats
This page is part of the EmailServer article.
Maillog analysis with AWStats
Maillogs are mighty important when troubleshooting or to check if your server is properly configured for its load.
The problem with raw maillog is that it quickly becomes hard to get any time of big picture view of what's going on: you can easily trace individual connections, but you won't know how many emails you send every day, how many megabytes are being moved or how many DNS errors your system is having.
AWStats is a generic log analysis package that offers a nice web interface and can be made to analyse almost any type of logs. It's easy to extend too and quite simple to install.
I don;t recommend using the RPM from the AWStat website has it has been packaged differently for Fedora and if you use automatic update, your config will change. Just use yum
to get it and make a virgin configuration file:
# echo alias awstats_updateall.pl=\
"'awstats_updateall.pl -awstatsprog=/var/www/awstats/awstats.pl'" \
>> ~/.bashrc
# cd /etc/awstats/
# cp awstats.model.conf awstats.mail.conf
The second line creates an alias for the awstats_updateall.pl
to include the path to the awstats.pl
script. This is necessary as the Fedora version of AWStats moved the files to other locations but some scripts still expect them to be at the default one.
Then edit the /etc/awstats/awstats.mail.conf
file (don't forget to change the mail.example.com to your server hostname):
LogType=M
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
SiteDomain="mail.example.com"
LevelForBrowsersDetection=0
LevelForOSDetection=0
LevelForRefererAnalyze=0
LevelForRobotsDetection=0
LevelForWormsDetection=0
LevelForSearchEnginesDetection=0
LevelForFileTypesDetection=0
ShowMenu=1
ShowSummary=HB
ShowMonthStats=HB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=0
ShowHostsStats=HBL
ShowAuthenticatedUsers=0
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowSessionsStats=0
ShowPagesStats=0
ShowFileTypesStats=0
ShowFileSizesStats=0
ShowBrowsersStats=0
ShowOSStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1
Now make AWStats update its database regularly from a cron job (crontab -e
), for instance, every 2h for mail analysis and every 3h for the web logs analysis:
00 */3 * * * /var/www/awstats/awstats.pl -update -config=localhost.localdomain
Then we can construct the database right now so we can access it for the remaining of our configuration:
If there are errors, double-check that you entered all the correct options in the configuration file above. If the script says it can't find awstats.pl
, make sure your ~/.bashrc
file contains the alias we defined above (there should be no spaces on either side or the =
sign) and that you have logged off and back in to ensure the aliases were read (the ~/.bashrc
file is read each time you login).
AWStats Integration in Apache
So now we have AWStats installed and updating its database regularly but what we still need is to display the stats in a convenient way.
The default installation should have added the following lines to your /etc/httpd/conf.d/awstats.conf
file. to make this set up a bit more secure, we're going to allow access only from web browsers on our LAN and maybe from our other office at 212.87.250.3
, all others will get an Access Denied page instead:
ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
DirectoryIndex awstats.pl
Options ExecCGI
Order deny,allow
Deny from all
Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
</Directory>
Now, to access AWStats, you need to point your web browser to
http://mail.example.com/awstats/awstats.pl?config=mail
.
However, if you configured SquirrelMail in a similar way as we did, this will probably not work as we have been rewritting our web requests to ensure people were using HTTPS.
Our solution, amongst many possible, was to add a subdomain mailstats.example.com
pointing to our mail server and add the following rewrite rules in the /etc/httpd/conf.d/awstats.conf
:
<Virtualhost *:80>
ServerName mailstats.faiveley-fareast.com
DocumentRoot "/var/www/awstats"
Alias /awstats/icon/ /var/www/awstats/icon/
ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats>
DirectoryIndex awstats.pl
Options ExecCGI
order deny,allow
deny from all
Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
</Directory>
RewriteEngine on
RewriteLog "/var/log/httpd/rewrite_log"
RewriteLogLevel 4
RewriteCond %{REQUEST_URI} ^/mail/?$|^/?$ [NC]
RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
RewriteRule (^.*$) http://%1/awstats/awstats.pl?config=mail
RewriteCond %{REQUEST_URI} ^/web/?$ [NC]
RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
RewriteRule (^.*$) http://%1/awstats/awstats.pl?config=localhost.localdomain
RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} ^(mail\..*)$ [NC]
RewriteRule (^.*$) https://%1 [L,R]
RewriteCond %{HTTP_HOST} ^webmail\.(.*)$ [NC]
RewriteRule (^.*$) https://mail.%1 [L,R]
</Virtualhost>
Requests to the web server wil be transformed as such :
- http://mailstats.example.com or http://mailstats.example.com/mail
-> http://mailstats.example.com/awstats/awstats.pl?config=mail - http://mailstats.example.com/web
-> http://mailstats.example.com/awstats/awstats.pl?config=localhost.localdomain
Now, providing we created the mailstats sub-domain, we can access both our mail and web statistics without having to remember those long URLs.
Extending AWStats
AWStats by default will provide quite a bit of useful information, but it will also ignore some that it won't understand, like how much spam or viruses were found.
How to integrate this information into the stats collected by AWStats will be developped in the near future.
Resources
- http://awstats.sourceforge.net/ : AWStats home
- http://awstats.sourceforge.net/docs/awstats_faq.html#MAIL : AWStats configuration for maillogs
- http://httpd.apache.org/docs/2.0/vhosts/ : Apache virtual hosts
- http://httpd.apache.org/docs/2.0/rewrite/ : Apache's Mod-rewrite documentation
< TroubleShooting | EmailServer
Great article, i hope advanced articule could be released soon. Bye