How to Realize Rsyslog Centralized Logging in Linux?

Syslog is a traditional log management tool on Linux, and rsyslog is its improved version. Many systems have replaced syslog with rsyslog for log management.
Alibaba Cloud Simple Application Server:  Anti COVID-19 SME Enablement Program
$300 coupon package for all new SMEs and a $500 coupon for paying customers.

Use rsyslog to record logs
Many applications currently support syslog for logging

• docker

docker run --log-driver=syslog --log-opt syslog-facility=local0 --log-opt tag=logTag

• systemd

# example.service
[service]
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=example
SyslogFacility=local0

Start a rsyslog server
Generally, the rsyslog package included on the Linux server has a service function, but it is not enabled by default.

If you want to enable the log server function, you need to modify the /etc/rsyslog.conf file.

The changes are as follows:

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# Uncomment the above two lines, as shown below, you can open UDP port 514 for log collection
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# Uncomment the above two lines, as shown below, you can open TCP port 514 for log collection
module(load="imtcp")
input(type="imtcp" port="514")

If you only need UDP or TCP, just uncomment the corresponding two lines.

The client log is sent to the remote server
If you need to send logs on server A to server B, you need to enable the rsyslog receiving log function on server B.
At the same time, modify the /etc/rsyslog.conf file on the A server.

Examples of changes are as follows:

# The normal log is saved as shown below
# [facility].[level] [destination]
# Example: mail.warn /var/log/mail.warn

# Three sending methods
# UDP prefix is @ @192.168.1.2:514
# TCP prefix is @@ @@192.168.1.2:514
# RELP prefix is: omrelp: :omrelp:192.168.1.2:514

# Send all logs of local0 to 192.168.1.2 via UDP, as on the machine (the default port number is 514)
local0.* @192.168.1.2
# Send the info level log generated by the user application to the machine 192.168.1.2 in TCP mode, the port number is 514
user.info @@192.168.1.2:514

# Distinguish by the syslog tag, and send the log prefixed with myapp- to the remote server
:syslogtag, startswith, "myapp-" @@192.168.1.2
# Filter the content of the message, and send the sns_log to the remote server
:rawmsg, contains, "sns_log" @@192.168.1.2
# Discard messages containing sns_debug
:rawmsg, contains, "sns_debug" ~

Note that this file is sequence dependent.

Here rawmsg is the message field, see rsyslog message field for details

In addition to startswith and contains, there are other comparison keywords: isequal, regex, contains_i, ereregex, isempty

The server receives remote logs
The server also needs to modify /etc/rsyslog.conf to receive remote logs.

Change example:

# Define a template named customformat, add log time to the information
$template customformat, "%TIMESTAMP:::date-mysql% %FROMHOST-IP%%msg%n"
# Define the name of the log file, according to year, month, and day
$template DynFile, "/var/log/%$programname%%$year%%$month%%$day%.log"
# Write the information containing the sns_log flag in the rawmsg (msg) log to the log file defined by DynFile
:rawmsg, contains, "sns_log" ?DynFile;customformat
# This means to discard the information containing the sns_debug flag, and it is generally added to avoid multiple log files recording duplicate logs
:rawmsg, contains, "sns_debug" ~

Forward non-syslog logs to the remote server
It is assumed that there is a helloworld application, and the generated log is recorded in the file /var/log/helloworld.log.

First, modify the /etc/rsyslog.conf configuration file on all machines in the application.

# Load the imfile module
module(load="imfile" PollingInterval="5")
# Specify the log file path and the parameters of tag, severity, facility
input(type="imfile" File="/var/log/helloworld.log" Tag="helloworld" Severity="error" Facility="local0")
# Send the helloworld application log to the remote server
:programname, contains, "helloworld" @192.168.1.2

Nginx logs are sent to rsyslog
Nginx (1.7.1+) can send logs directly to rsyslog, as shown below:

# nginx.conf
# server {
error_log syslog:server=192.168.1.1 debug; # Send the debug type log to the machine 192.168.1.1
access_log syslog:server=[2001:db8::1]:12345,facility=local7,tag=nginx,severity=info combined;
#}