Running a log server

Why run a log server?

Although logging events to files stored locally allows us to organise our logs, and keep historic records for auditing purposes, it rapidly becomes impractical to manage the storage and analysis of log files spread over anything other than a very small number of hosts.

Another good reason to keep security related logs on a central server is that the first thing a competent attacker would usually do after gaining access to a compromised machine is delete any trace of their attack from the log files. This, fairly obviously, helps prevent either the system administrator or automated log analysis software from spotting the intrusion. If these logs are stored on a centralised server this task will be much more difficult to accomplish.

As if the previous two reasons were not enough in themselves keeping all the logs files on a central system makes their analysis considerably easier as the log analysis software only needs to be installed on a single machine.

TCP or UDP

Like many network capable services which use the IP protocol for network communications the syslog-ng daemon is capable of communicating over either the connection oriented TCP protocol or the connectionless UDP protocol. Each of these protocols has advantages and disadvantages which may encourage, or in some cases even force, you to use one over the other, or even both.

TCP

The TCP protocol is usually the best choice for clients which are also running the syslog-ng logging daemon. It offers a connection oriented service which guarantees that no log messages will be lost in transit and allows the client to detect any failure of the connection and attempt to re-establish it.

The default syslog port for UDP is 514 which does not conflict with anything. When using the TCP protocol port 514 may also be used by the rsh remote shell service. This service should never be running in a production environment however as it does not use passwords!

 

A log source to receive log messages sent over the network using the TCP protocol can be declared in the configuration file as shown below.

source s_network_tcp { tcp(); }

While this will create a socket which listens on the default TCP port for the syslog service, which is 514, using the default options, which allow for a maximum of 10 simultaneous connections which will not be closed in response to a SIGHUP, these defaults can be overridden by specifying the new options as arguments to the tcp source driver as shown below.

source s_network_tcp { tcp( port(5014) keep-alive(no) max-connections(100) ); }

If your log server will be receiving logs from more than ten clients, which is the default maximum number of connections, then remember to ensure that you set the max-connections option accordingly.

 

UDP

As UDP offers no guarantee that messages will be delivered some log messages may be lost when logging over this protocol. The UDP protocol should, therefore, probably only be used when backward compatibility with the original syslog daemon is required or if connection reliability is already ensured by the underlying network transport such as an IP over IP tunnel.

A log source to receive log messages sent over the network using the UDP protocol can be declared in the configuration file as shown below.

source s_network_udp { udp(); }

As with the tcp log source we examined in the previous section the udp source can have additional parameters configured as shown in the example declaration below.

source s_network_udp { udp( port(5014) ); }

As the UDP protocol is connectionless it does not support the keep-alive or max-connections options.

TCP wrappers

We mentioned in the installation instructions that the syslog-ng service could be installed with support for TCP wrappers. If you enabled this use-flag when installing be sure to add entries to the appropriate files as shown below remembering to also change the IP addresses as needed.

The entries shown above will allow access to the syslog-ng service to any machines on the 10.x.x.x network and the loopback interface while denying access to all other systems.

Network logging options

The syslog-ng daemon provides some options which can be applied globally that are applicable to a network server configuration. These are described below.

chain_hostnames(no)

By default when receiving log messages over a network the syslog-ng daemon will append the name of any other syslog-ng servers the log message passes through. Setting this to no will cause just the sending host name to be recorded.

keep_hostname(yes)

When set to yes the name of the originating log host will be used when a message has passed through an intermediate server instead of the name of the intermediate server. This has no effect on directly received messages.

check_hostname(yes)

This parameter, if set, will cause the syslog-ng daemon to check the host name in all received log messages for invalid characters and remove them if found. This should be set to help ensure that an attacker cannot exploit the log server by sending a malformed host name when using the $HOST macro.

use_dns(yes)

If you are interested in performing reverse DNS lookups on the source of log messages, and recording this information to be used as a simple test of the validity of the logging host name, this parameter should be set to yes.

dns_cache(yes)

To avoid the syslog-ng daemon performing a DNS lookup on receipt of every log message this option should be enabled when the use_dns option is in use.

dns_cache_size(100)

This option controls the size of the DNS cache. You should always ensure that there are sufficient available entries to ensure that all the DNS lookups for the expected number of log clients can be cached, probably with a few spare.

dns_cache_expire(600), dns_cache_expire_failed(60)

These options control the time that a successful DNS lookup and a failed DNS lookup will remain in the cache. This example provides an expiration time of ten minutes and one minute respectively.

use_fqdn(no)

Normally the syslog-ng daemon just records the host name in the log files. If this option is set to yes then the fully qualified domain name of the sending host will be used instead.

use_time_recvd(no)

If you have reason to believe that the clocks of the client machines could be tampered with then this option could be set to yes so that the time that the log message was received is recorded in the logs instead of the time that the log message claims to have been sent. A better idea is to record both the sent and received times to compare for signs of tampering.

A set of reasonable example options is given below. They should be combined with the options which we examined in previous sections.

options
{
    chain_hostnames(no);
    keep_hostname(yes);
    check_hostname(yes);
    use_dns(no);
    dns_cache(no);
    use_fqdn(no);
    use_time_recvd(no);
}

If you enable the use_dns option remember to also configure the dns_cache and dns_cache_size options, or configure a local caching DNS server, to avoid a potential denial of service attack as a reverse DNS lookup will be performed for every log message or connection.

 

Using macros to further organise log files

In the previous sections we described how to configure syslog-ng to receive log messages from remote systems. If we combine this with the configuration which we have developed so far we will be able to log these messages to files categorised using their facility and level. In this section we shall examine some of the other macros which the syslog-ng daemon provides to allow us to organise our log files.

By host name

With the current configuration, rather unfortunately, all the log entries from our client machines will be stored in the same sets of files when what we probably want is to add another level of directories to the hierarchy so that these logs are further categorised by the name of the originating host. Fortunately the syslog-ng daemon provides macros called $HOST, which is expanded to the short host name, and $FULLHOST, which is expanded to the fully qualified domain name, which can be used for exactly this purpose.

The example below shows how the $HOST macro can be used to store logs in a directory structure based on the name of the remote log host as well as the date of the log entry.

destination d_mailerr      { file("/var/log/remote/$HOST/mail/$DATE/error.log"); };

As sent or as known

Earlier in this chapter we briefly mentioned that there were some security implications attached to assuming the veracity of any information we receive from a remote log host. For that reason the syslog-ng daemon provides an additional set of macros which can be used to prevent this kind of attack.

When working with dates and times these macros have the same name as normal simply prefixed with R_ and result in the current time of the log server being used at the moment of receipt. If you wish to force syslog-ng to use the time contained in the log message then you can use the macros prefixed with S_ instead.

The example below shows how the previous example could be rewritten using these macros to ensure that the correct time is used when rotating the files.

destination d_mailerr      { file("/var/log/remote/$HOST/mail/$R_DATE/error.log"); };

When we work with host names we also have to take care that they only contain values which we expect. The most effective method of achieving this is to use the use_dns option which we examined earlier. This will cause a reverse DNS lookup to be performed for every log message which is received and the result of that lookup to be used as the $HOST and $FULLHOST variables. You can obtain the host name that was sent by the log client by using the $HOST_FROM and $FULLHOST_FROM macros.