Log rotation

Why rotate my logs?

Having log entries separated by category and priority and stored to different files as appropriate is a good start but there is still a problem with the long-term scalability of our solution. More specifically, the more time that passes the larger our log files will become.

This can lead to a whole host of problems, and not just when it comes to locating log events recorded over a particular time frame in a given log file. If we are intending to perform automated log analysis, and we probably should be on a production server, how will we know which log entries to process? How will we stop our log files consuming all the available space on the log partition?

Automated log rotation is the answer to all these problems and more.

Rotation methods

Using the logrotate application

Traditionally on UNIX systems, and by implication Linux systems, log rotation has been performed by the, rather obviously named, logrotate application. It is usually run from a cron entry once per day whereupon it will examine its configuration files for the list of files it is to rotate and the rotation policy and then move, compress, delete, or email, the files accordingly.

We shall examine the use of the logrotate application in a later section when we look at rotating logs generated by services other than the syslog-ng daemon. We shall also provide a more detailed analysis of the shortcomings of this method, as well as offering some potential solutions.

 

Using syslog-ng macros

Another method of performing log rotation, and the one we shall examine here as this is a guide to system logging with syslog-ng, is to use the built-in macros provided by the syslog-ng daemon. These macros allow the various parameters of a log message, including the time and date it was generated, to be used to make up part of the path to the log file and are therefore perfectly designed for automated log rotation.

Rotating logs using syslog-ng macros

As we mentioned in the previous section the syslog-ng daemon provides a number of macros which expand to various elements of the log entry and can thus be used to perform log rotation. A brief description of the most commonly used of these macros is given below.

$DATE

This macro expands to the short form of the date according to the current system locale. On a UK system that would produce dates such as 30-01-07 while on a US system it would produce dates such as 01-30-07.

$DAY, $MONTH, $YEAR

These macros expand to the two digit day, month, and year, respectively. The expression $DAY-$MONTH-$YEAR is therefore equivalent to the $DATE expression above, at least on a UK system.

$ISODATE

This macro expands to the date in ISO format. Continuing the example from above this would be expected to expand to 2007-01-30 in all locales. For some reason, however, it in fact expands to an ISO date time structure such as 2007-01-30T14:55:33+0000 which makes it virtually useless.

$WEEKDAY

This macro expands to the three letter name of the day of the week that the log entry was generated. It would therefore expand to "Tue", without the quotes, on our example date.

$HOUR, $MIN, $SEC

These macros expand to the two digit hour, minute, and second, respectively. Other than the $HOURmacro they are probably of little practical use for log rotation but they are included here for completeness.

These macros can, amongst other things, be placed in the path specifier for file destinations as shown in the following example. This would cause the syslog-ng daemon to create files with the current date as their name, grouped in specified directories by log level, with the .log extension.

destination d_maildebug    { file("/var/log/mail/debug/$DATE.log"); };
destination d_mailinfo     { file("/var/log/mail/info/$DATE.log");  };
destination d_mailwarn     { file("/var/log/mail/warn/$DATE.log");  };
destination d_mailerr      { file("/var/log/mail/error/$DATE.log"); };

Alternatively we could place the $DATEmacro so that log files are grouped by date first, then priority level, as shown in the example below. This would cause the syslog-ng daemon to create files named level.log grouped in directories named with the current date.

destination d_maildebug    { file("/var/log/mail/$DATE/debug.log"); };
destination d_mailinfo     { file("/var/log/mail/$DATE/info.log");  };
destination d_mailwarn     { file("/var/log/mail/$DATE/warn.log");  };
destination d_mailerr      { file("/var/log/mail/$DATE/error.log"); };

Compressing and deleting logs

While the above configuration entries will allow us to record log entries in files organised according to the date of their occurrence as well as their facility code and priority level it is still missing some of the more important functionality of the standard syslog and logrotate combination. Most importantly it lacks functionality to archive log files in a compressed form and ultimately delete them when they are over a predefined age.

Unfortunately the syslog-ng service does not contain the required functionality to perform either of these functions. Thankfully it is a trivial task to write a simple script to replicate this functionality. This script can then be run from a cron entry in the same way as the logrotate program.

For your convenience a simple bash script which performs these functions is provided in the appendix.