Rsyslog for VM Level System Log¶
In this example, assuming you are a old fashion linux admin, you want to use rsyslog
to collect system audit log.
Quick Start¶
To get started, launch an Amazon Linux EC2 and ssh into it.
Amazon Linux already have rsyslog installed and running as a service when start up.
Verify rsyslog is already installed as a service:
systemctl list-unit-files --type service | grep rsyslog
Here’s some commands to run, stop, restart, check status:
# start
systemctl start rsyslog
# stop
systemctl stop rsyslog
# restart
systemctl restart rsyslog
# check status
systemctl status rsyslog
The rsyslog configuration file locates at /etc/rsyslog.conf
, You should back up the default configure and then customize it with confidence:
# view rsyslog config
# to update configure, you can back the default file, change it,
# and restart service with ``systemctl restart rsyslog``
cat /etc/rsyslog.conf
# backup rsyslog config file
sudo cp /etc/rsyslog.conf /etc/rsyslog.config.bkp
By default, rsyslog will write different type of logs to different files in /var/log
directory, most of system log are stored in /var/log/messages
file:
# browse log directory, by default, log files locate at /var/log directory
# carefully read /etc/rsyslog.conf file to understand each file are storing what type of logs
# most of common system log are stored in /var/log/messages
ls /var/log -l
# view last 10 line
sudo tail -n 10 /var/log/messages
# monitor new coming lines as live-stream
sudo tail -f /var/log/messages
Again we should be aware that rsyslog is designed for system log. It WON’t log your command line input output history, message sent to stdout/stderr that generated from your application. But you do can do some customization to enable that.
Now let’s generate some logs to see if rsyslog captures it
Example.1 Manually Generate System Log and Sent to rsyslog¶
There’s a built-in shell command interface allows you to send log to syslog, it is logger
(https://linux.die.net/man/1/logger)
# send dummy log to rsyslog
logger -t myapp Hello World
# view the log
sudo tail -n 10 /var/log/messages
Example.2 Send log to rsyslog From Your Application¶
We can write a simple python script called rsyslog-example-app.py
using Python Standard Library logging (https://docs.python.org/3/library/logging.handlers.html#sysloghandler)
# -*- coding: utf-8 -*-
# content of rsyslog-example-app.py file
import sys
import logging.handlers
logger = logging.getLogger()
logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
syslog_handler = logging.handlers.SysLogHandler("/dev/log")
syslog_handler.setLevel(logging.INFO)
syslog_handler.setFormatter(formatter)
logger.addHandler(syslog_handler)
stream_handler = logging.StreamHandler(sys.stdout)
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
msg = "Hello Amazon"
logger.info(msg)
Then generate some log:
# generate dummy log
python rsyslog-example-app.py
# view the log
sudo tail -n 10 /var/log/messages
Note:
by default (take a look at your
/etc/rsyslog.conf
file), rsyslog are listening the/dev/log
. These two line are commented out by default:$ModLoad imudp.so $UDPServerRun 514That’s why we have
syslog_handler = logging.handlers.SysLogHandler("/dev/log")
is in our python code.If you want your rsyslog using UDP and listen to default port 514, you can simple
systemctl stop rsyslog
, then uncomment the two lines above in/etc/rsyslog.conf
file, and thensystemctl start rsyslog
. Then change your python code tosyslog_handler = logging.handlers.SysLogHandler()
. Thelogging.handlers.SysLogHandler()
uses UDP 514 by default.
Example.3: Log your Command line input output history¶
Your can log the command line input output history with rsyslog as well. Just Follow this post https://askubuntu.com/questions/161935/how-do-i-log-all-input-and-output-in-a-terminal-session. The idea is send a copy of your shell history to rsyslog by adding some customization code in your .bashrc
file.
Reference¶
Rsyslog Official: https://www.rsyslog.com/