Mail delivery filtering using Courier maildrop

What is mail delivery filtering?

Mail delivery filtering is the name given to the process of filtering mail as it is delivered to a mailbox. This filtering mechanism can be utilised to perform a wide range of post delivery tasks using information gathered from the message headers or body. Such tasks may include activities such as moving mail to a folder or marking a message as spam in a way which can be detected by email client applications such as Outlook and Evolution.

Installing additional software

Additional database configuration

As the maildrop delivery method uses significantly more resources than the default virtual delivery method used by Postfix for virtual mailboxes the first task is to create the database framework required to allow Postfix to use a different transport for each account or alias based on the contents of a field in the relevant table.

To accomplish this end we must begin by creating a table which we shall use to store the details of the configured transports which may be used for mail delivery by our users and aliases. Once the table is created we can populate it with a suitable set of default values to enable use of both the existing virtual delivery mechanism as well as the maildrop delivery mechanism we shall be creating. Suitable SQL commands to perform these tasks are shown below. As you can see this table is extremely simple and consists solely of the key value we shall be using for each transport along with the name of the transport as named in the /etc/postfix/master.cf configuration file.

postgres=# CREATE TABLE transports ( 

postgres(#    id          CHAR        PRIMARY KEY, 

postgres(#    transport   VARCHAR(64) NOT NULL 

postgres(# ); 

NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "transports_pkey" for table "transports" 

CREATE TABLE 

postgres=# INSERT INTO transports(id, transport) VALUES('V', 'virtual'); 

INSERT 0 1 

postgres=# INSERT INTO transports(id, transport) VALUES('M', 'maildrop'); 

INSERT 0 1 

Now that we have created a table to store the details of our transports, and populated it with some suitable records, we are able to make the required modifications to the existing table we have created to store our mailbox records. To allow us to specify a transport for each mailbox we shall and add a field which will reference the key field in the transports table we just created. As the maildrop delivery method demands considerably greater system resources than the Postfix default virtual transport we shall specify this as the default value for our mailboxes to avoid wasting resources unnecessarily.

postgres=# ALTER TABLE mailboxes ADD COLUMN transport_id CHAR NOT NULL DEFAULT 'V'; 

ALTER TABLE 

postgres=# ALTER TABLE mailboxes ADD FOREIGN KEY (transport_id) REFERENCES transports(id); 

ALTER TABLE 

The next phase of the process is to create a view to consolidate this information into a form more easily queried by the Postgres server. Once the view is created we shall also assign suitable permissions to allow read only access to the mail server processes and full access to the administrator. As always suitable SQL commands are presented below.

postgres=# CREATE VIEW transport_maps AS 

postgres-#   SELECT m.username, d.name AS domain, t.transport 

postgres-#     FROM mailboxes m, domains d, transports t 

postgres-#       WHERE m.domain_id = d.id AND 

postgres-#             m.transport_id = t.id; 

CREATE VIEW 

postgres=# GRANT ALL ON transports, transport_maps TO mail_admin; 

GRANT 

postgres=# GRANT SELECT ON transport_maps TO mail_server; 

GRANT 

Finally we can change the transport used by some, or in the example below all, of our mailboxes to use the new maildrop transport method we are configuring here so that we have at least one account which we can use to test the newly added functionality.

postgres=# UPDATE mailboxes SET transport = 'M'; 

UPDATE 3 

The UPDATE statement in the above example will change the transport for all users of all domains. As the maildrop delivery mechanism requires significantly more resources than the virtual delivery mechanism this may not be desirable. If this is the case a suitable WHERE clause should be added to appropriately limit the updated records.

 

Additional Postfix configuration

As is often the case when configuring the Postfix server to offer additional functionality we need to provide details of the SQL database server, database, user and password as well as the required query to access the data. The example file below contains all the required information and as it is so similar to those discussed in previous sections shall not be covered again here.

hosts    = database
dbname   = mail
user     = mail_server
password = mail_server_password

query    = SELECT transport FROM transport_maps WHERE username = '%u' AND domain = '%d';

With the data access configuration completed we can now configure the Postfix server to make us of this data by adding the transport_maps directive shown below to our existing Postfix configuration file. To ensure that the correct rules are used for each recipient we shall also configure the Postfix daemon to deliver at most a single recipient at a time to a mailbox and also to deliver at most one item at a time to a mailbox.

maildrop_destination_recipient_limit = 1
maildrop_destination_concurrency_limit = 1
transport_maps = pgsql:/etc/postfix/pgsql/pgsql-virtual-transportmaps.cf

Before we can make use of this transport for delivering mail to our mailboxes there is one final task remaining. The maildrop transport needs to be defined so that the Postfix daemon knows how to use it. This configuration is located in the /etc/postfix/master.cf configuration file as shown below. A complete description of the syntax of the Postfix master configuration file is beyond the scope of this document however the relevant options should be fairly obvious from the example and should be suitable for most installations.

maildrop  unix  -       n       n       -       -       pipe
    flags=DRhu user=vmail argv=/usr/bin/maildrop
    -d ${user}@${nexthop} ${extension} ${recipient} ${user} ${nexthop}

Once the maildrop configuration is complete all which remains is to restart the Postfix server so that our configuration changes are used immediately. It would probably also be prudent to test that our mail server still performs as expected and can deliver mail using the existing virtual delivery method as well as the newly configured maildrop delivery method before creating any filtering code which may complicate matters.

The maildrop delivery method can now be tested as suggested by sending a message to a suitably configured mailbox to ensure that mail is delivered as usual before we create any filter rules. This mailbox must have already received at least one message using the virtual delivery method or the delivery directory will not exist.

 

Maildrop filtering configuration

Assuming that all went well so far and your test messages were delivered correctly we can begin to create a basic filtering infrastructure which can be built on to provide more functional configurations. As we mentioned in the above warning we lost some functionality with the switch from the default virtual delivery method to the maildrop delivery method so we shall also recreate this missing functionality in our system wide maildrop configuration.

As you can see from the example below we have broken our configuration into five logical sections. The first of these sections simply copies the variables which were passed to the maildrop application by the Postfix daemon into variables with more user-friendly names. The second block tests whether the home directory for this user, which will be the domain name they are registered with appended to the path of our mail store, already exists and creates it if not. The third block does the same for the user's default mail directory, creating it with the maildirmake command if required. The remaining two blocks include domain-wide and user-specific mail filter configuration files and are wrapped in exception blocks so that if either of these files does not exist mail filtering will continue.

EXTENSION=$1
RECIPIENT=$2
USER=$3
DOMAIN=$4

`test -e $HOME`
if ( $RETURNCODE != 0 )
{
    `mkdir -p $HOME`
}

`test -e $HOME/$DEFAULT`
if ( $RETURNCODE != 0 )
{
    `maildirmake $HOME/$DEFAULT`
    `chmod -R 0700 $HOME/$DEFAULT`
}

exception {
    include "$HOME/.mailfilters/.domain"
}

exception {
    include "$HOME/.mailfilters/$USER"
}

Rather annoyingly the maildrop filtering language insists that the opening curly-brace, which signifies the block starting after certain keywords, must appear in an inconsistent location. For example the opening curly-brace must appear on the following line for an if statement and the same line for an exception statement.

 

To enable testing of the domain-wide and user-specific mail filter configuration files we first need to create a directory to store such files. As we mentioned earlier these files are located in a subdirectory of the mail-store directory for their associated domain. Such a directory can be created using the command below replacing the example domain with whichever domain is being configured.

To demonstrate some of the other capabilities of the maildrop filtering system we shall implement a simple but very useful rule which can be used to mark emails which have already been flagged as spam, perhaps using the content filtering system presented in a later chapter, so that email clients such as Outlook and Evolution will be able to detect this without the need for additional client configuration. As you can see we are using the filename .domain so that these rules will be included for all users of this domain.

The filter starts with an statement specifying, as a Perl regular expression, the headers which should be matched by this rule. In this case anything starting with the string X-Spam-Flag: which then also contains the string YES in the same header line will be matched. The next line of interest associates the keyword Junk with this email using the predefined KEYWORDS variable. This is the line which actually performs the magic and tells IMAP clients that this message is in fact spam. The next section of interest attempts to deliver the mail to a directory called Spam and continues without error should this directory not exist.

if (/^X-Spam-Flag:.*YES/)
{
    KEYWORDS="Junk"

    exception {
        to $DEFAULT/.Spam/
    }
}

As a demonstration of user-specific filtering rules we shall create a rule for the alice account we used in our aliases examples earlier. This filter will deliver any mail from the Xen development lists to the Mailing Lists/Xen Development subdirectory of Alice's mailbox creating each directory level as necessary.

if (/^List-Id:.*xen-devel/)
{
    `test -e $HOME/$DEFAULT/.Mailing.Lists/`
    if ( $RETURNCODE != 0 )
    {
        `maildirmake $HOME/$DEFAULT/.Mailing.Lists`
        `chmod -R 0700 $HOME/$DEFAULT.Mailing.Lists`
    }
    `test -e $HOME/$DEFAULT/.Mailing.Lists/.Xen.Development`
    if ( $RETURNCODE != 0 )
    {
        `maildirmake $HOME/$DEFAULT/.Mailing.Lists/.Xen.Development`
        `chmod -R 0700 $HOME/$DEFAULT.Mailing.Lists/.Xen.Development`
    }
    exception {
        to $DEFAULT/.Mailing.Lists/.Xen.Development/
    }
}

Finally we create an example rule for the bob account to store all incoming mail in a folder called Inbox so that it can be easily distinguished from other mail. This can be particularly useful for simple clients such as PDAs which may otherwise attempt to retrieve all subfolders of the mailbox.

`test -e $HOME/$DEFAULT/.Inbox`
if ( $RETURNCODE != 0 )
{
    `maildirmake $HOME/$DEFAULT/.Inbox`
    `chmod -R 0700 $HOME/$DEFAULT/.Inbox`
}

exception {
    to $DEFAULT/.Inbox/
}