Setting Up a high performance IMAP server

The aim of this article is to describe how to configure a reliable and high performance email server for up to 200 employees.

The constraint here is not to send/receive millions of messages a day but to allow users to keep their emails on the server and use IMAP to retrieve them from wherever they are, and for however long they want to keep them.

This type of server is particularly suited for companies whose employees deal with a lot of emails, such as an Engineering, Project Management or a Law firm.

Objectives

The basic objective of this project is to enable authorised users to securely send, retrieve and manage lots of emails from anywhere.

Security is enforced from different angles:

  • Authorized use of email for sending, relaying or retrieving is only allowed for users who have an account on that system: for each operation, authentication is required.
  • Encrypted communications: whether sending, receiving or managing emails through IMAP, communication between the client and the server are encrypted. This is especially important for roaming users who may use unsecured wireless access points for instance.
  • Anti-spam protection to limit the risk of being exposed to phishing scams and to identify good mail from bad.
  • Executable attachment protection to ensure users will not receive potentially dangerous attachments in their emails (most executable attachments are forbidden, if you need to send an executable, zip it first). This attachment protection can be extended to control the type of documents sent through emails, for instance, you could ban avi or jpg files based on their extension.
  • Anti-virus protection to remove email containing viruses as they arrive at the server. Note that it is recommended to keep an additional level of protection in the form of an up-to-date antivirus programme on each client machine.

Additionally, the technical objectives are:

  • enabling users to send any email through the server (with a size limit if you want),
  • enabling roaming users to send email through the server securely from the Internet (using encryption),
  • enabling users to check their email through a secured web interface, wherever they are (using HTTPS),
  • allowing up to hundred of thousands of email messages to be stored in an IMAP folder without noticeable slow down when operating on messages (deleting, reading, getting headers)
  • easily add or remove users from the system while not allowing them to access the server from a terminal.
  • keeping the system, virus database and spam detection up-to-date

Implementation

For this project, I chose the following implementation:

  • postfix will be the Mail Transport Agent, receiving and sending emails and dropping them in each user mail boxes. saslauthd will enable email to be relayed through the server only for users who have an account on the machine.
  • dovecot will be our POP3/IMAP server, giving users access to their emails.
  • squirrelmail will be our web interface. it uses IMAP to access the local mail boxes.
  • amavisd (Amavisd-New) will be our filtering system, marshalling emails beetween postfix and spamassassin -our spam detector- and clamd, the ClamAV antivirus daemon.

Assumptions

In this document, the FQDN? (Fully Qualified Domain Name) of the server is also its hostname mail.example.com and its domain name is thus example.com.

I assume that you have setup a MX DNS record for mail.example.com and that mail sent to joe@example.com will be attempted to be delivered to the server we're setting up.

This setup is meant for a company wanting to host its own email server, so the focus is on single domain only. If you want to manage multiple domains, you will have to adapt this setup.

Note: if you need to setup your domain name and take control of your email, I suggest that you take an account with ZoneEdit: the first 5 zones are free, and extra services are quite cheap. You have total control over you domains, subdomains, and can of course create Mail eXchange records. They also have a service to buffer your email should your server be off-line. when things return to normal, any email they kept will be delivered as usual.

In the whole installation and configuration process, I assume that you are logged in under the root account.

Installation

The initial installation process is not critical here. Simply avoid installing too much stuff that you don't need (like X, Gnome or KDE and anything graphical) and limit the number of services that run on the server. It's easy enough to add missing packages afterwards.

Buy your hard-drives in double or triple and mount your partitions as RAID1 or RAID5 arrays: this is quite critical as it means that you won't lose everything if one of the drives fails. I won't discuss backup solutions for this system, but you should have one. In the future, I plan to describe a mail to database solution that could be used as a specific mail backup/archiving/retrieval system.

When deciding your disk partitions, allow a rather large one that you will mount as /mail and format as ext3.
Make it as large as you can. You could also build an LVM on top of that partition if you believe that you could run out of disk space. With LVM, you can easily increase or decrease your available space by simply adding new partitions to the Volume.

During the installation process, make sure that you choose Postfix in the available packages. Fedora defaults to using sendmail, so we'll have to fix that later.

After you've rebooted, use yum to update your system.

Check partition indexes

By default, Fedora Core and RedHat/CentOS systems use indexes when creating ext3 partitions, this is good and is required for the partition where the emails will be saved.

Why are indexes so important you ask?
Well, since we're going to use maildirs, each email will be saved as a file. Without indexes, it will take an increasing amount of time for the kernel to locate a file in a busy mailbox. The kernel limit that issue by building fast indexes so it can easily find the file it needs.

It doesn't hurt to double check that our partition supports indexes. Let's suppose that your /mail partition is /dev/hda3:

tune2fs -l /dev/hda3 | grep features

You should see a reference to dir_index.
If not, then you can modify your partition without losing data:

umount /dev/hda3
tune2fs -O dir_index /dev/hda3
e2fsk -fD /dev/hda3
mount /dev/hda3

If your system complains about not finding tune2fs, get it from yum install e2fsprogs.

This bit was taken from the Dovecot site.

Users

There are plenty of different ways to authenticate users for mail access.
By default though, Postfix, Dovecot and saslauthd -the services we use here- use the standard unix user accounts to validate users and find out their home directory.

Using LDAP, virtual users, MySQL databases, NIS, SMB or any other scheme is of course possible but in our case I wanted to keep things simple and allow this server to later become say a file server as well without too much hassle. Changing the authentication scheme can be necessary if you're managing virtual email accounts with lots of different domains (if you're and ISP for instance).
In our case, there was nothing much to gain.

Having said that, there are a few of drawbacks to using standard user accounts:

  • by default, these accounts are login accounts, meaning that users can log onto the machine at will. While they are not supposed to have enough credentials to wreak havoc in the system, it's still a potential security risk.
  • Whenever you create a new user account, its home directory is filled with local configuration files that we don't need (such as bash profiles, or emacs config files)
  • The default user account is created in /home but we want them in /mail instead.

I will assume that all our users will not need to login onto the machine by default. If you need such users, you can still override these settings.

Edit the /etc/default/useradd file:

# useradd defaults file
GROUP=100
HOME=/mail
INACTIVE=-1
EXPIRE=
SHELL=/bin/nologin
SKEL=/etc/skelmail
CREATE_MAIL_SPOOL=no

Note: set the Create Mail Spool to no otherwise the user inbox becomes a mbox file created under /var/spool/mail/ (which we don't want).

Now create the skeleton directory:

# mkdir -p /etc/skelmail/{cur,new,tmp}
# chmod 700 -R /etc/skelmail

Now, whenever we use the useradd command, the user will be added to the system without the ability to login and his mail directory will be automatically created.

Note: It's very important that the email folder is not world-accessible: Postfix will otherwise refuse to write any email in it as it would be a security hazard. That's why we set it to chmod 700.

Should you need to add normal login users accounts, you can override the default settings of adduser on the command line as such:

# adduser -m -k /etc/skel -s /bin/bash -d /home/susan susan

Will add the normal login user susan to the system.

Switch to Postfix

We need to switch from the sendmail, which Fedora uses by default, to postfix.
This stuff is pretty specific to RedHat's way of thinking: they wanted to make email management totally transparent so that it didn't really matter which MTA (Mail Transport Agent) you were using.

To switch, do the following:

# yum -t install postfix
# yum -t install system-switch-mail
# /usr/sbin/system-switch-mail

The first 2 yum installs may not be necessary if they are already on your system, but it won't hurt if you do it anyway (the -t switch tells yum not to complain if the packages are already installed).

You will be presented with a choice on screen, go for Postfix and press OK. After a few seconds, you should get a succinct report that the switch was completed.

Configuring Postfix

Postfix uses a few configuration files. The most important files are master.cf and main.cf and are located in /etc/postfix.

Let's first make a backup copy of these config files, just in case:

# cp master.cf master.cf.ORIGINAL
# cp main.cf main.cf.ORIGINAL

main.cf parameters

Now, edit the main.cf file and update the following definitions. Alternatively, you can use postconf -e "definition" on the command line for each line below if you don't want to edit the file by hand.

myorigin = $mydomain

This says that mail sent from your server will take the form xxx@example.com.

Note that by default, $myhostname and $mydomain are automatically derived from your machine's name. This name should be a Fully Qualified Domain Name (FQDM) like mail.example.com.
If that's not the case, you can change it editing the /etc/sysconfig/network file or you can declare the myhostname = mail.example.com and mydomain = example.com in the main.cf file itself.

mydestination = $myhostname localhost.$mydomain localhost $mydomain

Defines which domains you want to receive mail for. We should always allow the variations of localhost so the server can accept mail sent to itself, and $myhostname and $mydomain ensure that you will get mail sent to both mail.example.com and example.com.
If you want your server to accept mail sent to multiple domains (say myhome.com and loveme.org), add them to the mydestination list as well.

mynetworks_style = subnet

Allows people on the local network to be able to use the server to relay their emails. People from outside the subnet (outside of the IP addresses defined by your network's netmask, such as 255.255.255.0) will not be able to use the server to send email. This is safe, you never want unknown people from the Internet to be able to relay their mail through your server: it would only take a few minutes for your machine to become a spam hub.

relay_domains = $mydestination

Authorises people from the outside to send email that is supposed to be for us.

notify_classes = resource, software

Defines what sort of information should be sent to the postmaster when there is a problem. There are more options to that, but using too many could flood your mailbox.

relayhost =

Confirms we're not using any external relays as we want the server to deliver our emails directly to other servers. If your ISP doesn't let you send emails by yourself (some block port 25), then you can put their own email server there [mail.isp.com] (including the brackets). Any mail you sent through your server will be given to your ISP's email server for delivery. Note that this is not very reliable as ISP have usually no guarantee that your email will be delivered to its destination: you're in effect sending your mail through a black hole.

proxy_interfaces = 1.2.3.4

Is only needed if your server is not directly connected to the Internet but is for instance behind a firewall that uses Port Forwarding to redirect traffic to it on a local subnet (for instance, your server address is 192.168.0.1 or another reserved LAN IP Class). In that case, you have to tell Postfix what is the outside address of the mail server (replace 1.2.3.4 by whatever is your real IP). Note though that if you don't have a fixed IP, this can be a bit annoying and you may be better off with connecting the server directly to the Internet and using iptables as a good internal firewall.

inet_interfaces = all

Makes Postfix listen to all interfaces for email.

message_size_limit = 20971520

Limits the size of emails. Here we set it to 20MB which should be more than enough for most systems. It's a good idea to set a limit. I've have users trying to send 150MB emails to people who only had a dial-up connection (since delivery to the server from the local network is fast, people tend not to notice much the size of the emails they send).

masquerade_domains = $mydomain

Ensures that mail from other hosts being sent through the server gets rewritten with our domain name correctly appended. this means that if elise@accounts.example.com sends an email through the server, it will be rewritten as elise@example.com.
If you're accepting mail for multiple domains, you must add them to the list as well, separating them by a coma.

mail_name = MyOwnPostOffice

Optional and replaces the default name returned by Postfix. It's not a bad idea to replace the default string as it is par of the messages exchanged every time an email is being delivered. Potentially, it could allow someone to use that information to exploit a known security hole (the default string contains the full version number of Postfix).

home_mailbox = email/

If that directive is present, it will tell Postfix to deliver messages to the email/ folder inside the user's home, as in /mail/joe/email, instead of the default /var/mail/joe mbox file.
Note that the trailing / means that we want to use maildir instead of mbox. This is how we get to save our received emails in directories and files rather than in one single flat file that becomes cumbersome and fragile if there are too many stored emails in it.

Note: you do not need to create the directories: Postfix will do that for you if they don't exist.

SASL Authentication

As we've discussed before, ensuring that your server is locked down is vital if you don't want to become the next spam relay.
However, we must still ensure that we've got a flexible system that allows all our legitimate users to send email from wherever they are.

SASL is a way of authenticating users when they are trying to send mail. It uses a variety of methods and it's fairly flexible, at the expense of being simple.

To ensure proper SASL authentication, add the following to your main.conf file:

smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions =
  permit_mynetworks
  permit_sasl_authenticated
  reject_unauth_destination
  reject_invalid_hostname
  reject_unknown_sender_domain

The first line enables authentication for email being sent through the server from the outside (allowing roaming users to send email from the Internet).
the next directive ensures that Postfix will work around older mail clients with broken implementations of SMTP authentication.
Finally, smtpd_recipient_restrictions lists the steps that an email must pass before being accepted. There are a lot of different possibilities. I have used what I think is reasonable for our system.

For security reasons, Postfix runs as an unprivileged user, meaning that it doesn't have access to your password files.
To be able to authenticate users, it must rely on an external service, saslauthd.

Fortunately, this is already installed on Fedora and probably on most distributions as well. Just to be sure, do the following from the prompt:

# yum -t install cyrus-sasl

The only configuration that tells saslautd that is is needed for mail authentication is in the file /usr/lib/sasl2/smtpd.conf which is installed as part of the Postfix package.
This file contains simply two parameters:

pwcheck_method: saslauthd
mech_list: plain login

Note: on other Linux systems, this file may be missing or may be located under /usr/local/lib/sasl2/smtpd.conf instead.

Make sure you restart the salsauthd server so the changes can be taken into account, otherwise you won't be able to send email as they will fail during the authentication process:

Aliases

The minimum alias that must be set-up is for the postmaster who will receive errors and warnings issued by Postfix:

Edit /etc/aliases and modify the following:

postmaster: administrator
root: administrator

The administrator user must have been created and you should probably the one using that account regularly to check for issues.

Note: after every modification of the alias file, you must run postalias /etc/aliases to rebuild the database that Postfix will use. Your changes won't be taken into account if you don't do that!

To add more aliases, just add them to /etc/aliases:

Email aliases

I usually use a fair number of variation on employee's names as aliases so most misspelling in an email address still end-up going to the right person. You should of course only give one email address to your user, for instance, your company policy could be that emails should be in the form suzan.smith@example.com.

suzan.smith: susan
s.smith:     susan
suzansmith:  susan
ssmith:      suzan
smith.suzan  suzan

accounts:    boss
marketing:   boss
sales:       boss
joe.doe      boss
joedoe       boss

Don't forget to run postalias /etc/aliases to rebuild the aliases database for Postfix.

Virtual map

Aliases are ok, but not very powerful, especially if you're hosting multiple domains and want a mail sent to sales@example.com be redirected to john while a mail to sales@myhome.com should go to suzan.

In that case, edit the /etc/postfix/virtual text file instead of /etc/aliases and add an entry for each possible mail and where you want it to be dropped instead.

suzan.smith: susan@example.com
s.smith:     susan@example.com
suzansmith:  susan@example.com
ssmith:      susan@example.com
smith.suzan  susan@example.com

sales@example.com:   suzan@example.com
sales@myhome.com:    john@example.com

@example.com: john@example.com

Here, anything sent to any variation of Susan's email address will be redirected to Susan's account, regardless of the domain they were sent to: s.smith@example.com and s.smith@myhome.com will be dropped into Susan's mailbox.

For email sent to sales, Suzan will be the recipient for example.com and John for myhome.com.

Any other email sent to invalid addresses (one that does not have a defined account, alias or virtual entry defined on the system) at example.com will be sent to John. This is a catch-all definition, but it is generally preferable to use a separate account for it as it will receive all spam sent to the servers to accounts that do not exist.

After editing /etc/postfix/virtual, you must run postmap /etc/virtual to update the actual binary database used by postfix for fast lookups.

After any change it's also a good idea to run service postfix reload to make sure that postfix will re-read its configuration.

Dovecot for POP3 and IMAP

Dovecot is an excellent and flexible POP3 and IMAP mail server. Its performance is outstanding and it can be setup to manage large amounts of emails.
Docevot is part of Fedora Core/RedHat/CentOS and we don't have to do too much to make it work, although setting encrypted communications takes a bit of work.

If you haven't got it on your system, you can build it from source or just use the ubiquitous yum:

# yum install dovecot
# chkconfig dovecot --levels 235 on

First, we need to tell Dovecot where to find our emails: as you remember, we told Postfix to use maildirs in the user directories instead of the default mbox.

Edit the /etc/dovecot.conf file so that only the following lines are uncommented:

protocols = imap imaps pop3 pop3s
listen = [::]
ssl_cert_file = /etc/pki/dovecot/certs/dovecot.pem
ssl_key_file = /etc/pki/dovecot/private/dovecot.pem
login_dir = /var/run/dovecot/login
login_greeting = Welcome to oBlue.
mail_location = maildir:/mail/%u
mail_full_filesystem_access = no
maildir_copy_with_hardlinks = yes
protocol imap {
imap_max_line_length = 65536
login_greeting_capability = no
imap_client_workarounds = outlook-idle
}
protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
protocol lda {
postmaster_address = postmaster@example.com
}
auth default {
mechanisms = plain
passdb pam {
}
userdb passwd {
}
user = root
}
dict {
}
plugin {

Restart Dovecot (service dovecot restart) and test it right away, trying to get email from your test administrator account through POP3 and IMAP. Note that you must have sent email to that account first so Postfix could create the proper directory structure, otherwise you'll get and error when trying to access the IMAP folder that doesn't exist yet.

SSL for IMAP and POP

In the older article for Fedora Core 4 we used stunnel to encrypt our seucred POP and IMAP channels. You can access that article on the SecureAccess page where we still use stunnel for securing email we send to the server..

To change a bit, we're going to follow Dovecot standard configuration and allow it to use SSL directly for POP3S and IMAPS.

First, remove the existing cert files, we're going to replace them later:

# rm /etc/pki/dovecot/certs/dovecot.pem

Then edit the /etc/pki/dovecot/dovecot-openssl.cnf to look suitatble for your organisation:

[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type
prompt = no

[ req_dn ]
# country (2 letter code)
C=HK
# State or Province Name (full name)
ST=Hong Kong
# Locality Name (eg. city)
L=Central
# Organization (eg. company)
O=ACME Publishing Ltd
# Organizational Unit Name (eg. section)
OU=Main Office
# Common Name (*.example.com is also possible)
CN=mail.example.com
# E-mail contact
emailAddress=postmaster@example.com

[ cert_type ]
nsCertType = server

Now all we need is to run the small script that the dovecot people have nicely provided to generate the key.

   # /usr/libexec/dovecot/mkcert.sh

Note that you may need to change the path to the exact version on your system.

That's it, you should now be able to connect to the server after configuring your client to use SSL.

Installing Amavisd-New

Amavisd is a mail filtering interface that allows an MTA such as Postfix to talk to specialzed external add-ons such as SpamAssassin or ClamAV amongst others. On its own, it doesn't do anything useful, but it will make our anstispam and antivirus tools work well together.

It used to require some work to install Amavisd-New but it now has been included in most distributions, so installing it along with its numerous dependencies is as simple as:

# yum -t install amavisd-new

Configuring Amavisd-New

Configuring Amavisd-New is simply a matter of editing its /etc/amavisd.conf and change the following:

$max_servers = 5;
$daemon_user  = 'amavis';
$daemon_group = 'amavis';
$mydomain = 'example.com';
$MYHOME   = '/var/amavis';

Here we tell Amavisd-New that it will be running under the amavis user account, that its home directory is /var/amavis and we tell it our domain name as well.

$mydomain = 'example.com';
@local_domains_maps = ( [".$mydomain", "example2.com", "example3.com"] );

Replacing example.com with you real domain name.
If you are scanning mail for more than one domain, list them in @local_domains_maps.

$max_servers = 5;

This tells Amavisd-New to always fork 5 children waiting for our mails. This means that, at any time, we'll be able to process 5 emails simultaneously. If you don't have much RAM, you can lower this figure to 2, but you'll have to reflect that as well in the /etc/postfix/master.cf (see below) otherwise Postif will expect 5 processes to be available.

$sa_tag_level_deflt  = undef;
$sa_tag2_level_deflt = 6.31;

Ensures that we always get X-Spam-Status and X-Spam-Level in our email headers, whatever the spam score, that anything above a spam score (see SpamAssassin) will be flagged as spam.

$sa_spam_subject_tag = '[SPAM] ';

This is what will be shown in the subject of an email detected as spam.

$final_virus_destiny = D_DISCARD;

We will quarantine messages containing viruses so they are not delivered to the user (when our antivirus will be installed).

@bypass_virus_checks_maps = (1);

We also disable virus checking for now because we haven't installed the antivirus yet and Amavisd-New will not let mail through if this option is disabled and there is no Antivirus.

$final_banned_destiny = D_BOUNCE;
$banned_filename_re = new_RE(
  qr'\.[^./]*[A-Za-z][^./]*\.(exe|vbs|pif|scr|bat|cmd|com|cpl|dll)\.?$'i,
  qr'^application/x-msdownload$'i,                   
  qr'^application/x-msdos-program$'i,
  qr'^application/hta$'i,
  [ qr'^\.(rpm|cpio|tar)$'       => 0 ]
  [ qr'^\.(zip|rar|arc|arj|zoo)$'=> 0 ],
  qr'.\.(ade|adp|app|bas|bat|cmd|com|cpl|crt|exe|fxp|grp|hlp|hta|
         inf|ins|isp|js|jse|lnk|mda|mde|mdw|mdt|mdz|msc|msp|mst|
         ops|pcd|pif|prg|reg|scr|sct|shb|shs|vb|vbe|vbs|
         wsc|wsf|wsh)$'
ix,
  qr'.\.(mim|b64|bhx|hqx|xxe|uu|uue)$'i,
  qr'^\.(exe-ms)$',                       
);

Here we make sure that emails containing attachments with banned extensions get bounced. While we'll probably end up trying to bounce viruses and spam, it's better to use this option to let legitimate senders who are sending us banned attachements that their messages did not get through.

We now have a couple of choices: sending our spam to the user with the modified subject line, or quarantine the message by sending it to a special user account.

Basic setup: spam email sent to user

The most basic setup is to let the user deal with the spam. The good thing about that is that they can judge if there is a false positive, and can still separate the spam by using filters in their mail clients to automatically move marked spam to a local folder.
It is also preferable to do that in a newly setup system as it makes it easier to detect how good your spam catching is and tweak your system.

In /etc/amavisd.conf change the following:

$sa_kill_level_deflt = 10000;

This will make Amavisd-New ignore any action for spam scores below this figure, which is normally beyond the highest possible spam score.

Advanced setup: quarantine to a special account

If instead of cloggin the users with spam messages we want to move them to a special user account on the server, we need to modify the following in /etc/amavisd.conf:

$sa_kill_level_deflt = 6.31;
$final_spam_destiny = D_DISCARD;

Edit your /etc/aliases and append the following to send all notification emails to the existing administrator user account (see the previous DoveCot? section about creating and configuring this account).

virusalert  : administrator
spam.police : administrator

Again, don't forget to rebuild the aliases database with postalias /etc/aliases.

Configuring Postfix to use Amavisd-New

Now we need to let postfix know about Amavisd-New. Postfix communicates with external programs through TCP/IP ports. This method allows a clear separation of processes and their priviledges as each is running under its own account.

Edit the /etc/postfix/master.cf file and append the following:

smtp-amavis  unix    -    -    y    -    5    smtp
 -o smtp_data_done_timeout=1200
 -o smtp_send_xforward_command=yes
 -o disable_dns_lookups=yes
 -o max_use=20

127.0.0.1:10025 inet    n    -    y    -    -    smtpd
 -o content_filter=
 -o local_recipient_maps=
 -o relay_recipient_maps=
 -o smtpd_restriction_classes=
 -o smtpd_helo_restrictions=
 -o smtpd_sender_restrictions=
 -o smtpd_recipient_restrictions=permit_mynetworks,reject
 -o mynetworks=127.0.0.0/8
 -o strict_rfc821_envelopes=yes
 -o smtpd_error_sleep_time=0
 -o smtpd_soft_error_limit=1001
 -o smtpd_hard_error_limit=1000
 -o receive_override_options=no_header_body_checks

Note: if you reduced the number of amavisd processes ($max_servers) to launch, you should reflect that number in the first line above as well.

And edit /etc/postfix/main.cf and append this definition:

content_filter = smtp-amavis:[127.0.0.0]:10024

Amavisd-New White and Black lists

If you want to make sure that some addresses always get through and that some always get banned, you will need to create a white list and a black list.

# touch /var/amavis/white.lst
# touch /var/amavis/black.lst
# chown amavis.amavis -R /var/amavis/

You need to make Amavisd-New know about those files. Edit /etc/amavisd.conf and add:

@whitelist_sender_maps = read_hash("$MYHOME/white.lst");
@blacklist_sender_maps = read_hash("$MYHOME/black.lst");

Now simply add each email address you want to unconditionnaly allow (white list)/block (black list) on a single line inside the relevant file.

Testing

Follow the simple testing procedures listed on the amavisd website to make sure that postfix and amavisd are properly configured:

http://www.ijs.si/software/amavisd/README.postfix.html#basics_testing

The only error you may be getting is related to hte lack of antivirus, something we'll move onto now.

SpamAssassin

Spamassassin is a very versatile and complete spam fighting solution. It uses statistical techniques as well as external blacklist and can be configured to use add-on tools to make its detection more refined.

SpamAssassin uses a rating system whereby each email goes through a list of tests and the mail is flagged for every positive test, increasing the number of spam points it is allocated. Each test allocates a variable number points or fraction of a points depending on how useful and reliable it is at detecting spam.

Once the email has gone through all the tests have been performed an action is taken based on the total number of points: if the score is high enough, we're sure that this is spam, if not, we can let the message through.
There are a number of in-between actions that can be performed, and messages can be refused for deliver, dropped, deleted, quarantined, marked as SPAM in their subject line, etc.

To install SpamAssassin, just use yum:

# yum -t install spamassassin

Edit the /etc/mail/spamassassin/local.cf and add the following:

bayes_path /var/amavis/.spamassassin/bayes
auto_whitelist_path /var/amavis/.spamassassin/auto-whitelist
lock_method flock
trusted_networks 192.168.0.

The last line tells your server which local network it can trust. This should be set to the IP range of your internal network.

Now, make sure SpamAssassin will run when we boot:

# chkconfig --levels 235 spamassassin on
# service spamassassin start

Initialise the Bayesian database:

# sa-learn --sync

Test our config by running:

# amavisd debug-sa

If what you did above was done properly, you should see debug: using "/var/amavis/.spamassassin/user_prefs" for user prefs file in the middle of all those spewed by Amavisd-New (scroll back or use Shift+PageUp keys).

Testing antispam

Just send an email with the following in the body:
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

This is a standard fake spam signature used to test antispam software.

Your /var/log/maillog file should show something similar to:

Jan 23 15:23:12 white amavis[28345]: (28345-01) Blocked SPAM, MYNETS LOCAL [192.168.0.101] [192.168.0.101] <nkadesign@nkadesign.com> -> <test@nkadesign.com>, Message-ID: <4979704E.3020906@nkadesign.com>, mail_id: RKeqXrbI1RJJ, Hits: 998.56, size: 649, 273 ms

The email should be marked with such a high spam score that it will never reach its destination and it gets discarded.
What actually happens to such mail depends on the settings you chose in the configuration of Amavisd-New.

Baysian filtering

Spamassassin includes a powerful statistical analysis that can help toward refining the score given to emails passing through it.
Instead of depending on other people's definition of what is spam, the baysian filter can be trained to recognise what you consider as being spam.

The only drawback with baysian analysis is that it needs a set of good (ham) and bad (spam) emails large enough to be accurate, and you need to sort these emails and manually train spamassassin.
Spamassassin is able to be trained against each user's preferences, but it creates a burden on the user, requiring her to think about and properly pick out any spam and save it in a separate location.
I generally prefer a hand-off approach where users don't have to worry too much about that. One of the reasons being that training spamassassin against an improperly categorised mailbox could actually be counter-productive and allow more spam to be delivered (you always have to think that some users will just delete spam or, worse, leave it in their mailbox).

To avoid any aggravation, I created a simple MissedSpam folder in one of the IMAP mail accounts that I use. I then simply have to move any spam that made its way to an inbox into that folder.
Users can forward their spam as attachments and I have then to manually open and drop the spam into the mailbox. This is a bit time-consuming, but it ensures that I have a chance to evaluate the email and ensure that the spam box is really accurate and does not contain legitimate emails that could later be falsely detected as spam.
As time progresses, less an less spam should be able to make its way into the system.

To train spamassassin as to what is spam and what is ham, make sure you have enough segregated emails (between 150-3000) in each mailbox being trained then issue the following:

# sa-learn --spam --sync /mail/postmaster/.MissedSpam/cur/

That would train spamassassin to recognise spam better.
To function properly (at all) you also need to train for ham. Make sure that each mailbox you train against has no spam in it!

# sa-learn --ham --sync /mail/emily/cur/
# sa-learn --ham --sync /mail/john/cur/
...

Make sure that the database ownership has not been reclaimed by root:

# chown amavis.amavis -R /var/amavis

If you do this regularly the amount of spam that managed to get through should reduce.

ClamAV Antivirus

Clam AntiVirus is a totally free -in all senses of the word- antivirus scanner. It is released under the GPL and has seen a lot of activity in the recent past.
Its virus definitions are generally good and new virus definitions can be automatically downloaded several times a day.

ClamAV is available as a ready-made package.

# yum -t install clamd

If you prefer to install from source, have a look at my older article on ClamAV.

Edit /etc/clamd.conf and add/modify the following parameters:

LogFile /var/log/clamav/clamd.log
LogFileMaxSize 10M
LogTime yes
LogSyslog yes
PidFile /var/run/clamav/clamd.pid
TemporaryDirectory /var/tmp
DatabaseDirectory /var/clamav
LocalSocket /var/run/clamav/clamd.socket
FixStaleSocket yes
MaxConnectionQueueLength 30
MaxThreads 50
ReadTimeout 300
User clamav
AllowSupplementaryGroups yes
ScanPE yes
ScanELF yes
DetectBrokenExecutables yes
ScanOLE2 yes
ScanMail yes
ScanArchive yes
ArchiveBlockEncrypted no
MaxFileSize 50M

All other parameters should be commented out, in particular Example, TCPSocket. and TCPAddess.

Now we need to tell Amavisd-New to enable virus scanning. Edit /etc/amavisd.conf and comment out the following :

# @bypass_virus_checks_maps = (1);  <- This line must be commented out

Look for the following section in /etc/amavisd.conf and uncomment it:

@av_scanners = (
['ClamAV-clamd',
   \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.socket"],
   qr/\bOK$/, qr/\bFOUND$/,
   qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],
...

Automated update of Virus definitions

Edit the /etc/freshclam.conf:

#Example <- This line should be commented out

Now create a new crontab -e job to launch the updater:

05 * * * * /usr/local/bin/freshclam --quiet

Now freshclam will check for updates five minutes past every hour.

Note: Make sure that the name of the socket file clamd.socket matches the one used by clamd above!

Testing the AntiVirus

Let's restart our setup to check that everything works fine.

# service postfix stop
# service amavisd stop
# service clamd stop

# service clamd start
# service amavisd start
# service postfix start

Go to the eicar antivirus test site and download the eicar.com.txt test file.
This file is only a test signature and it should be recognised by all antivirus software as a 'virus' (so you may have to temporarily disable your desktop antivirus for the duration of the test).

Send an email with the eicar.com.txt as an attachment to a test email that you send through the server.
Look at your /var/log/maillog file and you should see something similar to:

Jan 23 15:13:01 mailserver amavis[28181]: (28181-03) Blocked INFECTED (Eicar-Test-Signature), MYNETS LOCAL [192.168.0.101] [192.168.0.101] <nkadesign@nkadesign.com> -> <renaud@nkadesign.com>, quarantine: virus-9p6fXda5rBjH, Message-ID: <49796DE5.80804@nkadesign.com>, mail_id: 9p6fXda5rBjH, Hits: -, size: 1069, 253 ms

SquirrelMail web interface

SquirrelMail is the webmail interface that comes preloaded with Fedora.
If it's not installed, just use yum install squirelmail to get it. You will also of course need Apache and make sure that your firewall has port 80 (http) and 443 (https) open.

Make sure Apache is running service httpd start. SquirrelMail is available out of the box from http://mail.example.com/webmail/.

By default, SquirrelMail is not accessible from https, and since our server will only be for mail, there is not need for appending the /mail at the end of the URL, so we need to fix those shortcomings.

First get the original SquirrelMail Apache config file out of the way:

# cd /etc/httpd/conf.d/
# mv squirrelmail.conf squirrelmail.OLD

Then edit the /etc/httpd/conf.d/ssl.conf and add the following between the existing <Virtualhost> tags, then append the URL rewrite rules:

<VirtualHost _default_:443>
  ...
  DocumentRoot "/usr/share/squirrelmail"
  ServerName mail.example.com
  <Directory /usr/share/squirrelmail>
    AllowOverride None
    Options ExecCGI
    Order allow,deny
    Allow from all
  </Directory>
  ...
</Virtualhost>

RewriteEngine   on
# Log the rewrites, just in case we need to debug (increase leve for verbosity)
RewriteLog      "/var/log/httpd/rewrite_log"
RewriteLogLevel 0

RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} ^(mail\..*)$ [NC]
RewriteRule (^.*$)   https://%1 [L,R]

RewriteCond %{HTTP_HOST} ^webmail\.(.*)$ [NC]
RewriteRule (^.*$)   https://mail.%1 [L,R]

Now make sure that Apache will start at boot and start the service for now:

# chkconfig --levels 235 httpd on
# service httpd start

Using Sub-domains

As recommended earlier, use a DNS provider such as ZoneEdit to manage your domain and sub-domains.
I often use sub-domains to alias an existing sub-domain; for instance, our server is mail.example.com, but I also create a webmail.example.com so users wanting to access their webmail can intuitively do so.

You should be able to access your email from anywhere securely through http://mail.example.com/.
Requests to the web server wil be transformed as such :

  • http://mail.example.com
    -> https://mail.example.com
  • webmail.example.com (whether http or https)
    -> https://mail.example.com

Note: https connections require a digital certificate registered with a known authority. The certificate is only valid for one website and one IP address and you need to pay for it. If you don;t have a certificate, your users will receive a warning when trying to access the site. You will have to tell them not to worry about that if you don't want or can't have a certificate (if you're using dynamic IP for instance). The certificate is only necessary to confirms that the site using is really who it pretends to be, it doesn't affect the fact that communications are encrypted.

Firewall considerations

Email services by definition need to be able to connect to the outside world. Unless you are content being able o exchange emails in you own little private LAN, it's likely that -in the minimum configuration- at least your SMTP server will be able to access the outside world to send emails.
A more realistic configuration -as the one we've been building here- would allow access to roaming users getting their emails from outside the private LAN and would allow messages to delivered to our SMTP server, postfix.

The following TCP ports are those used by mail services:

  • SMTP: 25 (Plain text transfers)
  • IMAP: 143 (plain text transfers)
  • POP3: 110 (plain text transfers)
  • SMTPS: 465 (Secure, SMTP over SSL)
  • IMAPS: 993 (Secure, IMAP over SSL)
  • POP3S: 995 (Secure, POP over SSL)
  • HTTP: 80 (For for webmail, plain text transfer)
  • HTTPS: 443 (For for webmail, Secure HTTP over SSL)

Now, depending on how your users are supposed to access their email, you will need to open the necessary incoming POP, IMAP and HTTP ports and/or their secured variants on your Internet-facing connection.
In any case, you will need to open port 25 so other servers can communicate with yours and deliver their messages. If you want roaming users to be able to deliver their messages securely through your server, port 465 must also be opened.

In a next chapter, we will see that it is fairly easy to secure your email channels using SSL and circumvent any port blocking that often ISP put in place.

Firewall implementation

Don't forget to disable the default firewall!

To avoid conflicts with your existing firewall, make sure that you disable the one provided by default with your distribution!
On a Fedora machine, inkove system-config-securitylevel if you're in the GUI or system-config-securitylevel-tui on the command line to diable it.

Having a proper firewall on your server is a must. Instead of relying on the default firewall configuration, I always opt to use the strong firewall rules scripts that allow better security and monitoring of the connections.

These rules scripts are available in the IP-Masquerade-HOWTO available from The Linux Documentation Project site.

Specific rules

In my firewall script, I use the following rules in the OUTPUT section of the script:

echo -e "      - Allowing access to standard Email ports"
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 25  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 110  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 143  -j ACCEPT
echo -e "      - Allowing access to Secured Email ports"
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 465  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 995  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 993  -j ACCEPT
echo -e "      - Allowing access to Web ports"
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 80  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 443  -j ACCEPT

If you are not using the same firewall script, replace the following variables:

  • $IPTABLES by /sbin/iptables
  • $EXTIF by your Internet-facing interface (eth0 for instance)
  • $EXTIP by your Internet IP address.
  • $UNIVERSE by 0.0.0.0

To ensure that the script is invoked at startup and whenever my internet connection drops, I saved the script as /etc/rc.d/rc.firewall and added that path to my /etc/rc.d/rc.local startup file and my /etc/ppp/ip-up.local file which is executed whenever the PPP connection restarts.

Make sure the script is executable and modifiable only to root and restart it:

# cd /etc/rc.d
# chmod 744 rc.firewall
# chown root.root rc.firewall
# ./rc.firewall

That's all you should need to make your email services accessible from the outside world without compromising your whole machine.
Remember that you don;t have to poke all these holes in your firewall; only open the ports for the services you really need.

Foreword

This article describes the necessary steps to set up stunnel to provide alternate secured ports for postfix and dovecot (SMTP and IMAP/POP).
This technique is useful but it is not necessary since you can achieve the same result just by modifying your firewall settings to redirect traffic from any port to the secured SMTPS, IMAPS/POPS ports already provided out of the box by postfix and dovecot.

You can directly proceed to the AlternateAccess article if you don't need to use stunnel.

Secure, encrypted email channels

With the proliferation of wireless hotpots, people have been able to enjoy a new freedom in accessing the web and their emails: airport lounges, coffee shops, public spaces.
While all this freedom is wonderful, public access points (and most private ones) usually don't use any form of encryption to protect your transfers.
This is especially an issue on publicly accessible networks where absolutely anyone can eavesdrop on your communication without you being the wiser. In some countries (including the US it seems), internet communications are routinely filtered, taped or analysed for whatever purpose is deemed just, too often without any accountability.

One of the reasons that email can be the weak link in your organisation security is that most of the time authentication connections by your email client are still sent in clear text, meaning that every time you get email, you advertise your username and password.
A lot of organisations only use a single username/password for each of their users and their access needs, meaning that the same pair is not only used for authorizing access to email accounts, but also to login into your PC, domain or Intranet...

Stunnel to the rescue!

There are many ways to secure your email connections through SSL: most servers have their own configuration details allowing them to work through an encrypted channel.

Here though, will will use Stunnel to provide secured channels to postfix@2 and dovecot@@ rather than configuring each of them individually.

Stunnel is also easy to use and can allow you to create ad-hoc secure tunnels for any arbitrary network service as well.

Installing Stunnel

If Stunnel isn't on your machine yet (whereis stunnel), just grab it from your rpm repository (yum install stunnel) or get the source code from stunnel.org.

To install from yum as usual:

# yum -t install stunnel

To install the source code:

# cd /usr/local/src
# wget http://www.stunnel.org/download/stunnel/src/stunnel-X.XX.tar.gz
# tar xzvf stunnel-X.XX.tar.gz
# cd stunnel-X.XX
# ./configure
# make
# make install

Note that if you are installing on a RedHat machine, you may need to modify the files tools/Makefile.am and tools/Makefile.in to replace any occurrence of nogroup with nobody instead.

Note as well that the default installation directory is /usr/local/bin. If you don't want this, use the --prefix option of configure (call it with --help to get more details about the available options).

Creating the private key and certificate

Certificate of Authority

Creating a Self-Signed Certificate is only half of the story: since they are not issued by a recognised authority your mail clients will popup an information message to the user. The only way to solve this is to register your server with a recognized authority such as Verisign.
This isn't necessary, but larger organisations may want to look into the details of providing this extra-security to their users as it ensures that your server is really who it claims to be.

To be able to encrypt the communication, Stunnel needs to have access to a secure and unique private key and a certificate that it can issue to the clients connecting to it.
the certificate will contain information about your organisation and a public key that will be used by to encrypt data that can only be decrypted by the server running stunnel.

This step is important: don't use any default key that comes with your system: the communication channels would still be encrypted, but since these keys are publicly available, your communications could be decrypted without too much hassle.

To create the necessary keys just do the following:

# cd /etc/stunnel
# openssl req -newkey rsa:1024 -keyout privkey.pem -nodes  \
         -x509 -days 365 -out cert.pem

You will be asked a number of questions regarding your organisation. These will be included in the certificate issued by Stunnel when contacted by clients.
Also make sure you use the FQDN (Fully Qualitified Domain Name) of your email server when asked about your server name (for instance, mail.example.com).

Note that the certificate will only be available for the number of days specified when creating it. You can make your certificate valid for as little or as long as you wish.

Now, Stunnel requires that both certificate be in a single file with extra blank lines after each certificate.
Like most encryption-related configurations, it also requires that the certificate have limited access.
After creating the proper key, we don;t need the original ones any longer.

# cat privkey.pem > stunnel.pem
# echo "" >> stunnel.pem
# cat cert.pem >> stunnel.pem
# echo "" >> stunnel.pem
# chmod 600 *
# rm privkey.pem cert.pem

That's it, Stunnel is ready to be configured:

Create the /etc/stunnel/stunnel.conf with the following entries:

cert = /etc/stunnel/stunnel.pem

[imaps]
accept  = 993
connect =  143

[pops]
accept  = 995
connect =  110

[smtps]
accept  = 465
connect = 25

What this does is simply tell stunnel to listen to ports 993, 995 and 465 and spit out the decrypted stream of data to ports 143, 110 and 25.
The cert line should not be necessary unless you compiled stunnel from source and it is expecting a different path (check what it could be by running stunnel version).

Now we can try to launch stunnel manually to check that it works:

# stunnel

That's all that should be needed. If you are getting error messages, then try to add the path to your stunnel.conf file. If there are any other errors, we won't see them and stunnel won't tell us whether eveything is fine or not.
To make sure everything is OK, have a look at the /var/log/secure log file, it should end with the following lines:

... stunnel 4.08 on i386-redhat-linux-gnu PTHREAD+POLL+IPv4+LIBWRAP with OpenSSL 0.9.7f
... 500 clients allowed

If you get an error saying that the binding address is already in use check that your protocols entry in the /etc/dovecot.conf file doesn't contain pop3s or imaps.

To Allow Stunnel to become a normal service like any other, we're going to install an init script for it.

A script is provided in the source code tree under tools/stunnel.init but I recommend you get my copy instead as I have modified it to be manageable with the chkconfig and service utilities.

Install the service and schedule it to run:

# cp stunnel.init /etc/init.d/stunnel
# chmod 755 /etc/init.d/stunnel
# chown root.root /etc/init.d/stunnel
# chkconfig --add stunnel

That's should be all. You can check that stunnel will start and stop by using service stunnel start or service stunnel stop.

Alternate Access

It is not uncommon for ISP to deal with the problem of spam and other email annoyances by forcing users to go through their own email services.
Most often, they will block the SMTP port 25, barring your roaming or external users or offices to directly connect to your mail server to deliver their mail.

Another species of more devious filtering seems to be common in places like China. The largest provider, China Telecom, seems to have strange ways of limiting (or filtering) access to IMAP and POP services when connecting to servers outside of China.
I'm not sure how and why, but access to these ports would be OK for w while, then would suddenly be blocked while all other traffic would still be available.

To solve these issues I'm discussing here 2 solutions:

  • opening alternate ports to our SMTP or POP and IMAP services.
  • offering alternative encrypted channels for STMP or POP and IMAP services.

The first one may be enough to circumvent the restrictions imposed by most ISP and will work fine as long as their filter only block the usual email ports 25, 110 or 143.
We use IP forwarding to poke a hole in the firewall and redirect any incoming traffic to the standard ports.

The second also requires that we poke holes in our firewall, but we'll just ask stunnel to listen to other ports as well as the standard SSL ones discussed in the SecureAccess article.

Forwarding traffic from one port to another

So let's implement the first solution.
I'll chose to arbitrarily use ports 725, 710 and 743 for SMTP, POP and IMAP.

As discussed in the Firewall chapter, I use the strong firewall rules scripts provided by the Linux IP Masquerade HOWTO.

In the FORWARD section of the script, I add the following entries:

echo "     - FWD: Aternate SMTP, POP3 and IMAP ports"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p tcp --dport 725 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A PREROUTING -t nat -p tcp -d $EXTIP --dport 725  -j DNAT --to $INTIP1:25
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p tcp --dport 710 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A PREROUTING -t nat -p tcp -d $EXTIP --dport 710  -j DNAT --to $INTIP1:110
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -p tcp --dport 743 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A PREROUTING -t nat -p tcp -d $EXTIP --dport 743  -j DNAT --to $INTIP1:143

Now restart the script (should be saved as /etc/rc.d/rc.firewall) and modify your email client to send and receive its mail using the new ports.

Secure alternate access

Again, we're going to use alternate ports 825, 810 and 843 as our arbitrary secure ports.

First, we poke holes for these ports in our firewall, so just edit /etc/rc.d/rc.firewall and add the following to the OUTPUT section of the script:

echo -e "      - Allowing access to Alternate Secured Email ports"
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 825  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 810  -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -m state --state NEW,ESTABLISHED,RELATED \
          -p tcp -s $UNIVERSE -d $EXTIP --dport 843  -j ACCEPT

Restart the script and add the following entries in the the stunnel configuration file /etc/stunnel/stunnel.conf discussed in the SecureAccess chapter:

[altimaps]
accept  = 843
connect =  143

[altpops]
accept  = 810
connect =  110

[altsmtps]
accept  = 825
connect = 25

Restart stunnel (service stunnel restart if you installed the init script for it) and modify your email clients to use SSL and the alternative ports.

Mbox to Maildir migration

When upgrading a mail server, you may be faced with a rather 'interesting' problem: migrating old user account in the classic monolithic mbox format to the more tolerant and performing maildir.

Mbox is a way to store emails in a single file. It's old, compatible with most mail applications, but as your mail store grows, it becomes harder to manage emails contained within.
Things start to become shaky if your mailbox is large, with the risk of corruption or connection time-out becoming likely.

Maildir on the other hand is newer and not supported by all systems. Basically it's just a way to store emails as files within a folder hierarchy. Its advantages are mostly better performance and tolerance in case of failure since you're more likely to lose just one email instead of your entire mailbox.

Postfix (MTA) and Dovecot (POP3 and IMAP server) both support the maildir format and can integrate with each other seamlessly.

Migration

It's nevertheless a great pain when you have to migrate existing mailboxes to the new format, on a new server for instance.

There is apparently only one tool that allows this type of thing: it's a small Perl script maintained by Philip Mak that is available as Perfect_maildir.

Just download the single file script and you are ready to convert, unless you're getting a long error message and in that case you may need to download the Date::Parse Perl module.

To install it, just enter:

# perl -MCPAN -e 'install Date::Parse'

If this is the first time you install a module, you will be asked a series of question. Just keep pressing Enter until you're asked about your geographic area. Make your selection of mirrors to download from and Perl will do the rest. Next time, you won't be asked anything.

If you have a handful of accounts to migrate, doing it by hand may be tedious, but it can still be faster than going through setting up an automated process.

I found useful to use my mail client to create IMAP accounts for every user I needed to migrate, then use their original mailbox layout to re-create the directories they had.

You can get the old mailbox layout by viewing the .mailboxlist file in the user's home.
Once each directory had been re-created (using your mail client), use perfect_maildir.pl to migrate the email for each folder, for instance:

# perfect_maildir.pl /mail/michael/email < /oldmail/var/mail/michael
# perfect_maildir.pl /mail/michael/email/.Drafts < \
                     /oldmail/home/michael/Drafts
# chown michael.michael -R /mail/michael/email

Would first move the inbox and then the old Draft mboxes to the new maildirs and ensure that the files belong to the user; this is particularly important if you're doing the migration as root.

If you need to migrate a lot of accounts, then you should use the migration tools below. They make use of perfect_maildir.pl, so you'll need that as well. I've included a copy of the script on this site just in case.

Just in case it may be useful to you, here is a small script I used to move each original mbox inbox to the new maildir one:

#!/usr/bin/perl
my $n = $ARGV[0];
$n =~ s/^\s+//;
$n =~ s/\s+$//;
die "No mailbox for $n\n"  \
    unless (-e "/oldmail/spool/$n" && -e "/mail/$n");
`mkdir /mail/$n/email/cur` unless -e "/mail/$n/email/cur";
`mkdir /mail/$n/email/new` unless -e "/mail/$n/email/new";
`mkdir /mail/$n/email/tmp` unless -e "/mail/$n/email/tmp";
print `perfect_maildir.pl /mail/$n/email < /oldmail/spool/$n`;
`chown $n.$n -R /mail/$n`;

You will need to modify the hard-coded paths to suit your configuration. In mine, I used the following:

  • /mail/<username>/email is the destination maildir folder.
  • /oldmail/spool/<username> is the old inbox mbox.

If you're setting up a new machine and your inboxes are on the old one, you can make files available to the new system through NFS and mounting the exported directories locally or simply copying using scp.

Alternatives

If your old email is sitting in a POP3 account at an ISP, you can use fetchmail to retrieve it.

Fetchmail is easy to use and I won't go into the details except that you could use a /root/.fetchmailrc file containing one line per account for each user. Note that this file must be chmod 600 for security reasons or fetchmail won't touch it:

poll mail.isp.com protocol pop3 user roger to roger password 123456 keep

The keep at the end will leave the mail on the old account just in case.

Another way, useful if you are using IMAP on the old account and have created IMAP folders, is to have both old and new account in your client email for each user and drag-drop files from one to the other after having recreated the same directory structure manually.

Email clients configuration

...ToDo...

Troubleshooting SASL authentication

To ensure that your authentication process works fine, we'll check what the server reports when we try to feed it a correct login.

Since logins are base64 encoded, copy and paste the following in a file that you call encode_sasl_plain.pl (don't forget to chmod 755 to make it executable):

#!/usr/bin/perl
use strict;
use MIME::Base64;
if ( $#ARGV != 1 ) {
   die "Usage: encode_sasl_plain.pl <username> <password>\n";
}
print encode_base64("$ARGV[0]\0$ARGV[0]\0$ARGV[1]");
exit 0;

Testing your Authentication Config

This section was inspired from the very complete book Postfix: the Definitive Guide from O'Reilly. Highly reommended!

Then use it to encode a username/password pair as it would be expected by the mail server for authentication. Here I use the existing administrator user (the account must exist on the system):

# encode_sasl_plain.pl administrator 123456

Then, talk to your mail server manually:

  1. # telnet localhost 25
  2. Trying 127.0.0.1...
  3. Connected to localhost.localdomain (127.0.0.1).
  4. Escape character is '^]'.
  5. 220 mail.example.com ESMTP MyOwnPostOffice
  6. EHLO test.faraway.com
  7. 250-mail.example.com
  8. 250-PIPELINING
  9. 250-SIZE 20971520
  10. 250-VRFY
  11. 250-ETRN
  12. 250-AUTH LOGIN DIGEST-MD5 PLAIN CRAM-MD5
  13. 250-AUTH=LOGIN DIGEST-MD5 PLAIN CRAM-MD5
  14. 250 8BITMIME
  15. AUTH PLAIN YWRtaW5pc3RyYXRvcgBhZG1pbmlzdHJhdG9yADEyMzQ1Ng==
  16. 235 Authentication successful
  17. quit
  18. 221 Bye

The only lines you will need to type are 6, 15, 17. The others are the server's responses.

Maillog analysis with AWStats

Maillogs are mighty important when troubleshooting or to check if your server is properly configured for its load.
The problem with raw maillog is that it quickly becomes hard to get any time of big picture view of what's going on: you can easily trace individual connections, but you won't know how many emails you send every day, how many megabytes are being moved or how many DNS errors your system is having.

AWStats is a generic log analysis package that offers a nice web interface and can be made to analyse almost any type of logs. It's easy to extend too and quite simple to install.

I don;t recommend using the RPM from the AWStat website has it has been packaged differently for Fedora and if you use automatic update, your config will change. Just use yum to get it and make a virgin configuration file:

# yum install awstats

#  echo alias awstats_updateall.pl=\
"'awstats_updateall.pl -awstatsprog=/var/www/awstats/awstats.pl'" \
>> ~/.bashrc

# cd /etc/awstats/
# cp awstats.model.conf awstats.mail.conf

The second line creates an alias for the awstats_updateall.pl to include the path to the awstats.pl script. This is necessary as the Fedora version of AWStats moved the files to other locations but some scripts still expect them to be at the default one.

Then edit the /etc/awstats/awstats.mail.conf file (don't forget to change the mail.example.com to your server hostname):

LogFile="perl /usr/bin/maillogconvert.pl standard < /var/log/maillog |"
LogType=M
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
SiteDomain="mail.example.com"
LevelForBrowsersDetection=0
LevelForOSDetection=0
LevelForRefererAnalyze=0
LevelForRobotsDetection=0
LevelForWormsDetection=0
LevelForSearchEnginesDetection=0
LevelForFileTypesDetection=0
ShowMenu=1
ShowSummary=HB
ShowMonthStats=HB
ShowDaysOfMonthStats=HB
ShowDaysOfWeekStats=HB
ShowHoursStats=HB
ShowDomainsStats=0
ShowHostsStats=HBL
ShowAuthenticatedUsers=0
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowSessionsStats=0
ShowPagesStats=0
ShowFileTypesStats=0
ShowFileSizesStats=0
ShowBrowsersStats=0
ShowOSStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1

Now make AWStats update its database regularly from a cron job (crontab -e), for instance, every 2h for mail analysis and every 3h for the web logs analysis:

00 */2 * * * /var/www/awstats/awstats.pl -update -config=mail
00 */3 * * * /var/www/awstats/awstats.pl -update -config=localhost.localdomain

Then we can construct the database right now so we can access it for the remaining of our configuration:

#  awstats_updateall.pl now

If there are errors, double-check that you entered all the correct options in the configuration file above. If the script says it can't find awstats.pl, make sure your ~/.bashrc file contains the alias we defined above (there should be no spaces on either side or the = sign) and that you have logged off and back in to ensure the aliases were read (the ~/.bashrc file is read each time you login).

AWStats Integration in Apache

So now we have AWStats installed and updating its database regularly but what we still need is to display the stats in a convenient way.

The default installation should have added the following lines to your /etc/httpd/conf.d/awstats.conf file. to make this set up a bit more secure, we're going to allow access only from web browsers on our LAN and maybe from our other office at 212.87.250.3, all others will get an Access Denied page instead:

Alias /awstats/icon/ /var/www/awstats/icon/
ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
    DirectoryIndex awstats.pl
    Options ExecCGI
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
</Directory>

Now, to access AWStats, you need to point your web browser to http://mail.example.com/awstats/awstats.pl?config=mail .
However, if you configured SquirrelMail in a similar way as we did, this will probably not work as we have been rewritting our web requests to ensure people were using HTTPS.

Our solution, amongst many possible, was to add a subdomain mailstats.example.com pointing to our mail server and add the following rewrite rules in the /etc/httpd/conf.d/awstats.conf :

NameVirtualHost *:80
<Virtualhost *:80>
    ServerName mailstats.faiveley-fareast.com
    DocumentRoot "/var/www/awstats"
    Alias /awstats/icon/ /var/www/awstats/icon/
    ScriptAlias /awstats/ /var/www/awstats/
    <Directory /var/www/awstats>
        DirectoryIndex awstats.pl
        Options ExecCGI
        order deny,allow
        deny from all
        Allow from 127.0.0.1 192.168.0.0/255.255.255.0 212.87.250.3
    </Directory>

    RewriteEngine   on
    RewriteLog      "/var/log/httpd/rewrite_log"
    RewriteLogLevel 4

    RewriteCond %{REQUEST_URI} ^/mail/?$|^/?$ [NC]
    RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
    RewriteRule (^.*$)   http://%1/awstats/awstats.pl?config=mail

    RewriteCond %{REQUEST_URI} ^/web/?$ [NC]
    RewriteCond %{HTTP_HOST} ^(mailstats\..*) [NC]
    RewriteRule (^.*$)   http://%1/awstats/awstats.pl?config=localhost.localdomain

    RewriteCond %{HTTPS} off
    RewriteCond %{HTTP_HOST} ^(mail\..*)$ [NC]
    RewriteRule (^.*$)   https://%1 [L,R]

    RewriteCond %{HTTP_HOST} ^webmail\.(.*)$ [NC]
    RewriteRule (^.*$)   https://mail.%1 [L,R]
</Virtualhost>

Requests to the web server wil be transformed as such :

  • http://mailstats.example.com or http://mailstats.example.com/mail
    -> http://mailstats.example.com/awstats/awstats.pl?config=mail
  • http://mailstats.example.com/web
    -> http://mailstats.example.com/awstats/awstats.pl?config=localhost.localdomain

Now, providing we created the mailstats sub-domain, we can access both our mail and web statistics without having to remember those long URLs.

Extending AWStats

AWStats by default will provide quite a bit of useful information, but it will also ignore some that it won't understand, like how much spam or viruses were found.

How to integrate this information into the stats collected by AWStats will be developped in the near future.

References

Comments
CongTuesday 19 December 2006, at 19:29 GMT+8 [X]
This is most complete detail configuration for email that I ever seen. I use sendmail instead of posfix but I think I will switch over base on your your document. I see that you put a lot of work on here. Thank for for the document. What software do u use to write this document? I'm thinking writing HOWTO for linux like this as well please email me at congngo at yahoo.com
Tilemahos ManolatosMonday 16 July 2007, at 15:00 GMT+8 [X]
Congratulations -- this is the most detailed and complete tutorial regarding this topic I've ever seen Many thanks
Enter your comment (no links allowed): Author:

Design by N.Design Studio, adapted by solidGone.org (version 1.0.0)
Powered by pmwiki-2.2.0-beta65