William LeFebvre

The flexible and often-overlooked syslog facility provides system administrators with fine-grained control over error messages. When used to its fullest, syslog can make the administrator's job much easier by organizing log information. A well-designed, centralized logging facility also can augment security by providing a virtually incorruptible logging facility.

Overview

Each message sent to syslog has a facility and a severity level associated with it. Some facilities, such as auth and mail, have predefined meanings. Some facilities are set aside for locally defined uses. Certain system applications and network devices can be configured to send syslog information using any facility. The local facilities can be used for services such as routers and TCP Wrappers.

The severity level is an indication of the importance of a given message. The lowest severity level is debug, intended for a daemon's debugging output. This is especially useful during the development phase of server applications but is used rarely in day-to-day operations. The next higher level is info, which is intended for messages of an informational nature. The severity levels continue to increase in importance (see Figure 1).

Configuration File Syntax

The way in which the syslog daemon handles messages is defined in the configuration file /etc/syslog.conf. The syntax is simple. Comments start with the traditional #. Each line in the configuration file defines one possible course of action. The line is composed of two parts separated by a white space. The first part is a list of facility and severity pairs separated by semicolons, for example mail.debug;kernel.info. The second part is a destination: a filename, a username, or a remote logging host. Some systems allow the specification of a post-processing filter in the destination as well. Most systems also allow * to indicate that a message is to be sent to all current users. Messages that relate to one of the facility/severity pairs in the left column will be logged to the destination named on the right. In the case of usernames, the messages are sent to the named users if they are logged in.

To be logged, the incoming message must have come from the named facility and must have a severity equal to or higher than the one listed in syslog.conf. For example, the pair mail.err will match any syslog message for the mail facility with severity err or higher. The asterisk can be used to indicate all facilities, so *.debug specifies all logged messages. The level none can be used to deselect a facility, so *.info;mail.none specifies all messages at or above info except for mail facility messages.

A Sensible Configuration

Chances are that the syslog.conf file shipped with your system is not thorough. Most versions I have seen put all messages into a single file, perhaps shunting off mail to a separate file. The flexibility and versatility of syslog is rarely utilized in the configuration file shipped with the system. Consider discarding it and starting from scratch.

Because the data is already separated by facility, you should store it on disk that way. Write a configuration file that stores data from each facility in a separate log file. Choose or create a specific directory just for log files (such as /var/log) and place the files there. Start with severity level info on all facilities. Figure 2 shows an example syslog.conf file that accomplishes this for several facilities. Examining log messages for authentication is as easy as looking at the auth log file. If you discover that one of the facilities has an overzealous logger, increase the severity level so the info messages are not kept. The next step is to determine which messages are important enough for recording on the console or for alerting any users. On a user's workstation, it may never be appropriate to log anything to the console.

Finally, an administrator in charge of a network of UNIX machines needs to centralize the logs. It is much easier to check for abnormalities on one machine than on 30. Set aside one host as your loghost. This should be a host with lots of available disk space in the partition where logs are kept (typically /usr or /var). Configure syslog on that host to suit your needs. Then change all other hosts to redirect logging information to loghost. This can be done with a single line in syslog.conf: *.debug @loghost. If you feel some messages need to be logged on the console, put in configuration lines for those. Your procedures for installing a new UNIX client should include replacing syslog.conf with your own version. Some systems (notably Solaris) try to provide centralized log messages through a single syslog.conf file usable on the log server as well as on the clients. This is accomplished by using the m4 preprocessor to decide whether the host is the loghost. I say don't bother--just use a different syslog.conf file on loghost.

Rolling Logs Over

If left unattended, log files slowly will grow without bounds. Most systems include a cron job to move the active log file periodically and start a new one. These cron jobs usually are tailored to accommodate the syslog.conf file distributed with the system. But if you write a new one, you will have to devise your own scheme for rolling over old log files.

For most installations, it is sufficient to hang on to syslog messages for a certain period of time, typically four to eight weeks, then simply remove them. This can be accomplished by shifting files once a week and removing files that are too old. Some systems use numbered suffixes (syslog.0, syslog.1, syslog.2, and so forth) to indicate the age of the file. I have found it is better to use a suffix that includes the date, such as syslog.970723. It also is better to place the archives in a separate directory, such as a subdirectory named old. This keeps the archives from getting in the way when troubleshooting an active problem.

Ideally, what we need is a shell script flexible enough to handle all these requirements: any number of log files, any time interval, and a specifiable destination. Such a script, called newlog, is presented in Listing 1. There are some important items to note about the script. It does not copy the active log file but instead moves the file and creates a new log file. This guarantees that no log messages will be lost while creating the archives because syslog will keep the old file open until it receives a HUP signal. Compression, if performed, is done after syslog is "tickled" with a HUP signal. The script relies on a date command that accepts formatting strings. Finally, the script itself does not remove old log files. Although it can be handled in this script, it also can be handled with a different cron job that simply uses find.

The first argument to newlog indicates the directory in which log files are kept. The remaining arguments specify the files in that directory that should be archived. The script assumes the archive is contained in a subdirectory called old. This can be changed easily or even made into a command-line option. The newlog script cannot be used to archive more than once a day because of the way it calculates the suffix. If you use a suffix that includes the current time as well as the date, newlog could roll over logs any number of times during the day.

An example usage is given by the two crontab lines in Figure 3. The first line rolls some log files once a week, the second line rolls over the mail log file once a day, and the last line removes any archived log files older than three weeks.

The syslog Safe

One intriguing technique I have used is the construction of a syslog safe. This is a machine dedicated to receiving and processing syslog messages. To prevent malicious hackers from covering their tracks, you can tightly protect and control the syslog machine. It acts much like a drop safe. Messages sent to the machine are recorded and become virtually untouchable and unalterable.

The best syslog safe provides no other form of remote access. It allows no remote logins, no sendmail, and no other network services. Access to the safe is gained only by logging in to the console directly. This can be accomplished by deconfiguring inetd, sendmail, rpcbind (the portmapper), or other network service daemons. Hackers cannot corrupt what they cannot access. This limits to syslog only the machine's vulnerability via remote access.

The truly paranoid can attach a hard-copy device to the syslog safe and modify syslog.conf to log all important messages to the printer's serial line. If the machine and its printer are secured behind a locked door, the chance of a syslog message being removed once it is logged is extremely small.

Once the syslog safe is built, you need to ensure it gets all the important security-related data. This means installing tools on your clients that improve the system's ability to syslog events. Free tools such as TCP Wrappers and logdaemon are available from ftp://coast.cs.purdue.edu/pub/tools/unix/. [For more information on TCP Wrappers, see "Trick Hackers With TCP Wrappers," September 1997, p. 41, and for more information on logdaemon, see "Logging Logins," October 1997, p. 27.]

Commercial tools also are available. If you use the syslog safe as your loghost, all client messages will end up there.

The syslog safe is not a cure-all. It cannot guarantee every message will be recorded safely. It is still susceptible to several methods of attack. But it is yet one more tool in the fight against the wily hacker.

Summary

syslog is underutilized at many installations. Its ability to centralize and categorize logs can vastly improve proactive management of a workstation network. A dedicated logging host, constructed as a drop safe, can improve the security of logged messages. In conjunction with tools such as logdaemon and TCP Wrappers, the syslog safe can provide intrusion detection that is difficult to alter.

The newlog Script
LISTING 1

#!/bin/sh
#
ARCHIVE="OLD"
if [ $# -lt 2 ]; then
echo "usage: newlog dir name"
exit 1
fi
#
# calculate suffix for current log files
suffix='date +%y%m%d'
#
# move to specified log directory
cd $1
shift
#
# iterate through each named log file
while [ x$1 != x ]; do
mv -f $1 $ARCHIVE/$1.$suffix
cp /dev/null $1
chmod 640 $1
shift
done
#
# tickle syslog to get it to use the new log files
kill -1 'cat /etc/syslog.pid'
#
# wait for syslog to switch over
sleep 5
#
# now compress the freshly archived logs
compress $ARCHIVE/*.$suffix

William LeFebvre has been banging on UNIX systems for 15 years and has been studying Internet technology for almost as long. He currently operates Group Sys Consulting and can be reached at wnl@groupsys.com.

syslog Severity Levels
FIGURE 1

Sensible syslog.conf
FIGURE 2

newlog Usage In crontab
FIGURE 3

This article first appeared in the November 1997 issue of UNIX Review.

Home | Top | Editor's Choice

Copyright © 2000 UnixReview.com ,UnixReview.com's Privacy Policy,
Comments: rreames@cmp.com
SDMG Web Sites: C/C++ Users Journal, Dr. Dobb's Journal, MSDN Magazine, Sys Admin, SD Expo, SD Magazine, UnixReview.com, Windows Developer's Journal