how to setup real-time email-notification for critical syslog events
a few weeks ago, i wrote a short article about the advantages of using syslog for all your logging needs. syslog is the standard logging solution for *nix platforms and integrates into virtually all application servers, network devices, and programming languages.
this statement logs the message "my pants are on fire" at a
next, configure syslog to log all
this line specifies a one second timeout for the bash builtin,
alternatively, if you'd like to turn this script into a log-running daemon that will sit in an endless loop and send out messages as soon as log statements arrive, remove the timeout line and surround the read statement with an while-true loop i.e.
if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.
it is often important for system administrators to get real time notification of critical events. unfortunately, it isn't immediately obvious how to do this in the syslog framework. in this article i show you step-by-step how to do this.
as usual, all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications.
the syslog advantage
one of the big advantages of syslog is the separation between the log request and the logging action. for example, a shell script might contain a log request like:logger -p local0.crit "my pants are on fire"critical level to a facility local0.
without changing your script, you can configure syslog perform some or all of these actions:
- write this to a local logfile
- log this to the console
- write this to a remote logging server
- send an email / sms in real time
let's focus on number 4. real-time notification is a good choice when your pants are on fire.
named-pipes
later versions of syslog have support for writing to named-pipes. a named-pipe is a special type of file that implements a simple fifo stream, allowing processes to talk to each other. we'll exploit named-pipes to implement real-time messaging between syslog and our mailer.in our example, we'll take all critical messages written to the local0 facility and (in addition to logging) send them to the mail recipient, fireman@example.com.
configuring syslog to write to a named-pipe
first, create a named-pipe for critical messages, for example:# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessagescritical messages written to the local0 facility to this pipe. add the following statement to your syslog.conf file.
local0.crit |/etc/syslog.pipes/criticalMessagessending out messages
the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i've included an example below, let's call it/usr/bin/syslogMailer:
#!/bin/bash
# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
# example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#
alertRecipient="fireman@example.com" # the mail recipient for alerts
TMOUT=1 # don't wait > 1 second for input
# process each line of input and produce an alert email
while read line
do
# remove any repeated messages
echo ${line} | grep "message repeated" > /dev/null 2>&1
if test $? -eq 1
then
# send the alert
echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
fi
donedaemon vs cron?
you'll notice that i've included the following line in the script:TMOUT=1 # don't wait > 1 second for inputread. the script therefore runs to completion after processing one batch of zero or more messages. this allows you to schedule it in cron to run, say, every 5 minutes with a statement like:
# m h dom mon dow command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1# process each line of input and produce an error message
while :
do
while read line
do
[...]
# send the alert
echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
done
donethe daemon approach is a little more efficient and sends out emails synchronously. it has the disadvantage that if your daemon terminates unexpectedly, alerts will stop until the daemon is restarted. the cron based implementation is arguably more robust in this regard. the cron approach also allows you to batch up notifications into n minute chunks. 5 minutes in our example cron file above.
tech blog
if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.
Regarding the OpenBSD
Regarding the OpenBSD problems...
Actually, as I discovered, syslog on the BSD's is a little different. Running syslog with -d allowed me to see that it was trying to execute the path after the pipe (|) in syslog.conf, so I'm actually calling up a script now similar to the one above to send me messages. I'm actually working out a way to have it "hold" on to messages allowing me to parse through it and remove what I don't want by email and forward it on, avoiding email bombs if something really gets messed up on the machine. Unfortunately, at this time it means I can't use the crontab which would have been preferred, again, to avoid being bombed by email in the event something goes horribly wrong.
Great blog on setting up
Great blog on setting up mailouts for syslog stuff. It's great for pretty basic needs and simple/single purpose devices like what I'm using it for (a low power mini-itx machine running as a router).
I've run into a bit of a snag, I can't quite figure out. I've setup everything as indicated here, and I understand pretty much all of what's going on, the problem I believe is with the script - and while I'm not much of a scripter, I know enough to know it's pretty basic and should work cross-platform, on bash/sh. For some reason, it won't work on my OpenBSD system. I can't get mailouts of desired log activity to work. when testing with logger the messages show up in the desired log files as per other syslog.conf settings, but nothing gets mailed out. In fact, nothing makes it to smtpd because /var/log/maillog has nothing relevant. When I run the script as such:
/usr/local/bin/syslogmailer < /etc/syslog.pipe/critmsgs
basically nothing is happening, and I have to ctrl-c the script to exit, which gives me this error (only on ksh though):
^Cksh: cannot open /etc/syslog.pipes/critmsgs: Interrupted system call
In the end, I'm not getting email, but nor is anything anywhere complaining, so I'm at a loss for troubleshooting. Ideas?
Regards,
Sandro
I would suggest using the
thanks Jonathan
thanks Jonathan
Does not seem to work on
Does not seem to work on Solaris 10. The bash script just hangs waiting for input.
that's too bad. i don't have
TMOUT If set to a value greater than zero, TMOUT is treated as the default timeout for the read builtin. The select command termi‐ nates if input does not arrive after TMOUT seconds when input is coming from a terminal. In an interactive shell, the value is interpreted as the number of seconds to wait for input after issuing the primary prompt. Bash terminates after waiting for that number of seconds if input does not arrive.alternatively, if you use the daemon approach that i outlined in the article, you don't use the read builtin anyway, so hopefully that should work for you.
Hi John. Thanks in advance
Hi John.
Thanks in advance for your script, however I wanna write you my poor experience with it:
TMOUT is ok in bash with Solaris 10. This parameter work fine. My appreciation is syslogd don´t write any message (i use notice) to pipe file. If it is regular file, syslogd write fine. Same behavior if script is daemon or cron program. I appreciate if you have any idea whats is wrong with script, syslog and Solaris 10. Thanks a lot. Regards.
Sergio
Sergio, sorry, I don't have
Sergio, sorry, I don't have a Solaris 10 environment available to test this on.
post new comment