System Logging Linux
System logs must be recorded for the auditing the system and troubleshooting the problems.
By Convention all logs are recorded in the /var/log directory.
Syslog message is handled by two services systemd-journald and rsyslog.
systemd-journald daemons collects log messages from the kernel, early stage of boot process, standard output and error of daemons as they start up and run and syslog. It write these message to structure journal of event. The syslog message is also forwarded by systemd-journald to rsyslog for further processing. Then rsyslog sorts the log by facility and priority and writes it to /var/log.
Some of the system log files:
- /var/log/message -> All common message are logged here, except security, email, corn jobs and those are purely debugging message.
- /var/log/secure -> The log files for security and authentication related message and errors.
- /var/log/mailog -> Logs mail-server related log.
- /var/log/cron -> the log related to cron jobs.
- /var/log/boot.log -> Message related to system startup are logged here.
Many programs and application uses syslog protocol to log event to the system log.
Each log messages is categorized by the facility( type of message) and a priority (severity of the message).
Facility: auth, authpriv,cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7.
Priority: debug, info, notice, warning, warn (same as warning),err, error (same as err), crit, alert, emerg, panic (same as emerg). The keywords error, warn and panic are deprecated and should not be used anymore
This is configured in /etc/rsyslog.conf. One can change the configuration in /etc/rsyslog.conf or place new file .conf suffix in /etc/rsyslog.d/ directory.
For more info look into man rsyslog.conf
Open /etc/rsyslog.conf to see content of it.
The logger command can send the log message to the rsyslog service. By default it send message to the facility user with priority notice (user.notice) unless specified otherwise with -p option
Example: Configure rsyslog to log all debug log message with any priority.
Step 1: Create a file named debug.conf in /etc/rsyslog.d/
$ echo "*.debug /var/log/debug_message.log" | sudo tee /etc/rsyslog.d/debug.conf
Step 2: Restart rsyslog service
$ sudo systemctl restart rsyslog
Now all debug log message will be recorded in the /var/log/debug_message.log. Let’s test it.
Tail the output of /var/log/debug_message.log
$ sudo tail -f /var/log/debug_message.log
From another terminal use logger to log a message.
$ logger -p user.debug “Debug test log message”
The systemd-journald stores the log structured, index binary file. This data include extra information about the log event. By default systemd journal is stored in /run/log and its contents are cleared after a reboot.
The journalctl command shows full system journal when run as root.
$ sudo journalctl
journalctl -n option shows last 10 log entry, it takes an extra parameter for how many last logr entries should be displayed.
$ sudo journalctl -n 20
Filter the output with priority
$ sudo journactl -p <priority>
journalctl -f outputs last 10 line and continues to output journal entries.
$ sudo journalctl -f
Limit the output to date range.
journalctl has two options ––since and ––until. Both takes the parameter in the format YYYY-MM-DD hh:mm:ss
$ sudo journalctl --since “2018-06-25” --until “2018-06-27”
There are fields attached to log entries that can only be visible when verbose output is turned on.
$ sudo journalctl -o verbose