Content filtering infrastructure using amavisd-new

Installing additional software

Before we can implement our content filter we first need to install the necessary software. As you can see in the example below we are only installing a single additional package, the amavisd daemon. This package requires so many dependencies that we have not listed them here. As before we have enabled the postgres use-flag so that we can use our database to store our content filtering preferences on a per-user basis.

Configuring amavisd-new

The configuration file for the amavisd-new package is extremely large and complex with a great many options which can be set to change the behaviour of the content filtering system as desired. As there are so many entries in this file which would need to be reproduced here before a working system would be configured we have only presented the changes required to each section of the default configuration file. After each section is a brief description of the configuration changes we have made sometimes accompanied by additional information.

# Section I - Essential daemon and MTA settings

$mydomain = 'hacking.co.uk';
$myhostname = 'mail.hacking.co.uk';

@bypass_virus_checks_maps = (1);
@bypass_spam_checks_maps  = (1);

$forward_method = 'smtp:[127.0.0.1]:10025';
$notify_method = $forward_method;

Section one is used to configure the basic daemon and MTA settings and as such requires very few changes. The most important changes in the above example are the middle grouping which disables spam and virus checking. These entries are required at this stage as we have not installed any spam or virus scanning software for the amavisd daemon to use. The remainder of this section is devoted to networking settings and should obviously be modified to reflect the host and domain names of your installation.

# Section II - MTA specific

$unix_socketname = undef

$inet_socket_bind = '127.0.0.1';
@inet_acl = qw(127.0.0.1 [::1]);

As you can see from the example above the second section of the configuration file contains settings related to the MTA we shall be using. Specifically these settings disable listening on a local UNIX socket, as we shall not be using this communications method, and instructs the daemon to bind an IP socket to the local loopback address. We also specify an ACL allowing communications with other processes via the loopback device only.

# Section III - Logging
$DO_SYSLOG = 1;

$syslog_priority = 'debug';

The third section contains the logging configuration for the amavisd daemon. Here we set the log level to debug to aid debugging should things not work as expected first time. Once configuration and testing is complete this should probably be set to 1 to avoid generating very large log files indeed.

# Section IV - Notifications, destiny and quarantine

$final_virus_destiny      = D_PASS;
$final_banned_destiny     = D_PASS;
$final_spam_destiny       = D_PASS;
$final_bad_header_destiny = D_PASS;

$virus_admin = undef;
$spam_admin = undef;

$mailfrom_notify_admin     = undef;
$mailfrom_notify_recip     = undef;
$mailfrom_notify_spamadmin = undef;

$QUARANTINEDIR = "$MYHOME/quarantine";
# $QUARANTINEDIR = "$MYHOME/quarantine";

$virus_quarantine_to      = undef;
$banned_quarantine_to     = undef;
$bad_header_quarantine_to = undef;
$spam_quarantine_to       = undef;

The final section of the configuration file we shall be modifying is section four which is devoted to configuring the actions to be taken in response to detecting a spam or virus infected message. As we currently have no spam or virus detection software installed and also have no policy mechanism configured the example above simply disables all notifications and other actions for the time being.

With the rather arduous configuration experience completed, for the moment at least, we can start amavisd and, assuming all goes well, add it to the default run-level using the commands below.

Reconfiguring Postfix

Now that we have configured the amavisd daemon to listen for messages to filter we need to configure the Postfix service so that it knows where to send incoming messages. As you can see from the example configuration below we have configured Postfix to use the LMTP protocol to pass messages to amavisd as well as limiting the number of uses to 20 so that any problems do not pose a permanent barrier to mail delivery. We have also specified an increased timeout period and instructed the Postfix server to send the xforward command.

amavisd   unix  -       -       n        -      2       lmtp
    -o lmtp_data_done_timeout=1200
    -o lmtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

Although this takes care of how we shall send messages from the initial Postfix instance to the amavisd daemon we still need to configure an additional instance of the Postfix service so that messages can be re-injected for delivery. This can be accomplished using the example below which, as you can see, overrides a great many of the options we have specified in our previous configuration example. As all the messages will have been subjected to these tests once already when they were received by the Internet facing Postfix instance and as there is no need to perform them again a significant performance improvement can be achieved by disabling them.

localhost:10025 inet n    -       n       -       -     smtpd
    -o content_filter=
    -o mynetworks=127.0.0.0/8
    -o smtpd_use_tls=no
    -o smtpd_sasl_auth_enable=no
    -o smtpd_delay_reject=no
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_end_of_data_restrictions=
    -o smtpd_restriction_classes=
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o local_header_rewrite_clients=
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters

With these changes in place we can reload the Postfix configuration and perform some basic testing before we make the final changes to route received mail through our content filter. This is possible as any mail sent to the system during testing will be delivered using the already working service we configured in previous sections. Only when we are certain that messages are being correctly delivered after passing through the filer will we make the changes required to route all incoming mail through the content filter.

Testing and activating the content filtering system

As mentioned in the previous section it is probably a good idea to test that the content filtering system is working in this very basic configuration before we route all our incoming email thought it. As before when testing the mail system we can use the telnet application to simulate a mail exchange from the Postfix service. As you can see in the example below we have to be slightly more strict with our syntax when testing the amavisd daemon than we did when we tested the initial configuration of the Postfix service.

Trying 127.0.0.1... 

Connected to localhost. 

Escape character is '^]'. 

220 [127.0.0.1] ESMTP amavisd-new service ready 

MAIL FROM: <someone@example.com> 

250 2.1.0 Sender <someone@example.com> OK 

RCPT TO: <spamcatcher@hacking.co.uk> 

250 2.1.5 Recipient <spamcatcher@hacking.co.uk> OK 

DATA  

354 End data with <CR><LF>.<CR><LF> 

To: spamcatcher@hacking.co.uk  

Subject: Test Message  

This is a test piece of mail.  

.  

250 2.0.0 OK: queued as D9F357BCD 

QUIT  

221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel 

Connection closed by foreign host. 

Assuming the above test worked and we received the test email we can be reasonably sure that our content filtering system is passing messages to the Postfix service we configured for mail re-injection and that they are being correctly delivered from there. We can now configure the Internet facing instance of the Postfix service to pass all received messages to the content filter with some degree of certainty that all will work as expected.

amavisd_destination_recipient_limit = 1
content_filter=amavisd:localhost:10024

The additional configuration entries in the above example are sufficient to instruct the Internet facing Postfix service to forward all received messages to the content filter, and thus to their final destination via re-injection to the additional Postfix instance we configured in the previous section.

At the moment however there is one "feature" of the system which may or may not be a problem on some installations. When mail arrives at the Internet facing Postfix service address-mappings will be performed and aliases expanded. This means that if an alias expands to several physical mailboxes any messages sent to this alias will be scanned multiple times by the content filter. This may be desirable in some cases as it allows each mailbox to set individual preferences for mail received by such an alias. In other cases the extra load placed on the system by this approach may be undesirable in which case the following modification should be made to the Postfix configuration so that the Internet facing instance does not perform address mappings and alias expansion.

smtp      inet  n       -       n       -       -       smtpd
    -o receive_override_options=no_address_mappings

Once the configuration is complete all which remains is to instruct the Postfix service to reload the configuration file so that it will start to use our newly configured content filter. This can be accomplished, as usual, with the following command.

Integrating amavisd with the database

Although the previous sections detailed the installation and testing of the content filtering framework we have still not enabled our users to do a great deal with it. To provide our users with full control over their content filter preferences we need to integrate the amavisd daemon with our existing database.

The first step in this process is to create a new table in our database to store the filter policies we shall create. A filter policy is simply a named group of configuration options which can be assigned to an alias or mailbox and will be applied when mail is received to that address. As you can see from the example SQL below there are a great many settings which can be configured to customise the behaviour of the content filter to suit a wide range of requirements. As these are all fairly self-explanatory, and as suitable default values are also supplied, we shall not be discussing them further in this section. A complete explanation can be found in the amavisd-new documentation.

postgres=# CREATE TABLE filter_policies ( 

postgres(#     id                        SERIAL PRIMARY KEY, 

postgres(#     policy_name               VARCHAR(32) NOT NULL, 

postgres(#  

postgres(#     virus_lover               BOOLEAN DEFAULT FALSE, 

postgres(#     spam_lover                BOOLEAN DEFAULT TRUE, 

postgres(#     banned_files_lover        BOOLEAN DEFAULT FALSE, 

postgres(#     bad_header_lover          BOOLEAN DEFAULT TRUE, 

postgres(#  

postgres(#     bypass_virus_checks       BOOLEAN DEFAULT FALSE, 

postgres(#     bypass_spam_checks        BOOLEAN DEFAULT FALSE, 

postgres(#     bypass_banned_checks      BOOLEAN DEFAULT FALSE, 

postgres(#     bypass_header_checks      BOOLEAN DEFAULT FALSE, 

postgres(#  

postgres(#     spam_modifies_subj        BOOLEAN DEFAULT FALSE, 

postgres(#  

postgres(#     spam_tag_level            FLOAT DEFAULT 0.0, 

postgres(#     spam_tag2_level           FLOAT DEFAULT 7.0, 

postgres(#     spam_kill_level           FLOAT DEFAULT 1000.0, 

postgres(#  

postgres(#     spam_dsn_cutoff_level     FLOAT DEFAULT NULL, 

postgres(#  

postgres(#     addr_extension_virus      VARCHAR(64) DEFAULT NULL, 

postgres(#     addr_extension_spam       VARCHAR(64) DEFAULT NULL, 

postgres(#     addr_extension_banned     VARCHAR(64) DEFAULT NULL, 

postgres(#     addr_extension_bad_header VARCHAR(64) DEFAULT NULL, 

postgres(#  

postgres(#     warnvirusrecip            BOOLEAN DEFAULT FALSE, 

postgres(#     warnbannedrecip           BOOLEAN DEFAULT FALSE, 

postgres(#     warnbadhrecip             BOOLEAN DEFAULT FALSE, 

postgres(#  

postgres(#     newvirus_admin            VARCHAR(64) DEFAULT NULL, 

postgres(#     virus_admin               VARCHAR(64) DEFAULT NULL, 

postgres(#     banned_admin              VARCHAR(64) DEFAULT NULL, 

postgres(#     bad_header_admin          VARCHAR(64) DEFAULT NULL, 

postgres(#     spam_admin                VARCHAR(64) DEFAULT NULL, 

postgres(#  

postgres(#     spam_subject_tag          VARCHAR(64) DEFAULT NULL, 

postgres(#     spam_subject_tag2         VARCHAR(64) DEFAULT NULL, 

postgres(#     message_size_limit        INTEGER     DEFAULT NULL, 

postgres(#     banned_rulenames          VARCHAR(64) DEFAULT NULL 

postgres(# ); 

NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "filter_policies_pkey" for table "filter_policies" 

CREATE TABLE 

Now that we have created a table to store our filter policies we can add a suitable default policy to this table. As we have no spam or virus filtering applications installed at this time this filter policy will simply bypass all the checks passing all mail to the destination without performing any filtering. As you can see from the example below we have named this filter policy "Bypass all checks" so that its purpose is obvious.

postgres=# INSERT INTO filter_policies(policy_name, bypass_virus_checks, bypass_spam_checks,  

postgres(#                             bypass_banned_checks, bypass_header_checks) 

postgres-#        VALUES('Bypass all checks', TRUE, TRUE, TRUE, TRUE); 

INSERT 0 1 

Once we have a suitable default policy defined in our filter_policies table we can modify the existing tables for mailboxes and aliases to include a field referencing this table using the SQL below. As you can see we specify a default filter policy so that all existing mailboxes and aliases will use this policy. We also add two foreign key constraints so that policies must exist before they can be selected and cannot be deleted if they are still in use.

postgres=# ALTER TABLE mailboxes ADD COLUMN filter_policy_id INTEGER NOT NULL DEFAULT 1;  

ALTER TABLE 

postgres=# ALTER TABLE aliases   ADD COLUMN filter_policy_id INTEGER NOT NULL DEFAULT 1;  

ALTER TABLE 

postgres=# ALTER TABLE mailboxes ADD FOREIGN KEY (filter_policy_id) REFERENCES filter_policies(id); 

ALTER TABLE 

postgres=# ALTER TABLE aliases   ADD FOREIGN KEY (filter_policy_id) REFERENCES filter_policies(id); 

ALTER TABLE 

With our modifications to the tables complete we can create a view to aggregate the policy information for mailboxes and aliases into a single convenient format which greatly simplifies the SQL queries which follow. As you can see this is a fairly complex join however processing requirements are surprisingly low as all joins are performed on indexed columns.

postgres=# CREATE VIEW policy_map AS SELECT f.id, m.username, a.address, d.name AS domain 

postgres-#   FROM filter_policies f 

postgres-#     FULL JOIN mailboxes m ON m.filter_policy_id = f.id 

postgres-#     FULL JOIN aliases a ON a.filter_policy_id = f.id, domains d  

postgres-#       WHERE m.domain_id = d.id OR a.domain_id = d.id; 

CREATE VIEW 

All that remains now is to grant suitable permissions on the newly created table and view so that the mail server account can read the required information. There is no need to explicitly grant permissions to the mail admin account as it is the database owner however feel free to do so if you wish.

postgres=# GRANT SELECT ON policy_map, filter_policies TO mail_server; 

GRANT 

Now that the database configuration is complete we can modify the configuration of the amavisd daemon so that it will query the database for the filter policy information. The following provides an example of the SQL datasource which will be used to query the database, which should be modified to suit your configuration, as well as the SQL query which should be suitable unmodified.

@lookup_sql_dsn = ( ['DBI:Pg:host=database;port=5432;dbname=mail', 'mail_server', 'mail_server_password']);

$sql_select_policy = 'SELECT f.*, %a AS id FROM policy_map p, filter_policies f WHERE p.id = f.id'.
                     '  AND p.domain = split_part(%a, \'@\', 2)'.
                     '  AND (p.username = split_part(split_part(%a, \'@\', 1), \'+\', 1)'.
                     '    OR  p.address = split_part(split_part(%a, \'@\', 1), \'+\', 1))';

When you are confident that the configuration is correct you can restart the amavisd content filtering service using the command shown below.