Linux Log Basics

First, we'll describe the basics of what Linux logs are, where to find them, and how they are created. If you already know this, please feel free to skip to the next section.

Linux system Logs

Many valuable log files are created automatically for you by Linux. You can find them in the/var/log directory. Here's what this directory looks like in a typical Ubuntu system:

Some of the most important Linux system logs include:

• /var/log/syslog or/var/log/messages stores all global system activity data, including boot information. Debian-based systems such as Ubuntu store them in/var/log/syslog, while RedHat-based systems such as RHEL or CentOS store them in/var/log/messages.
• /var/log/auth.log or/var/log/secure stores logs from pluggable authentication modules (PAM), including successful logins, failed login attempts, and authentication methods. Ubuntu and Debian store authentication information in/var/log/auth.log, while RedHat and CentOS store that information in/var/log/secure.
• /var/log/kern stores the kernel's error and warning data, which is especially useful for troubleshooting custom kernel-related failures.
• /var/log/cron stores information about cron jobs. Use this data to make sure that your cron job is running successfully.

Digital Ocean has a complete tutorial on these files that explains how Rsyslog creates them in common distributions such as RedHat and CentOS.

The application will also write to the log file in this directory. For example, common server programs like Apache,nginx,mysql can write log files in this directory. Some of these log files are created by the application itself, while others are created through the syslog (see below).

What is a Syslog?

How is the Linux system log file created? The answer is through the syslog daemon, which listens for log information on the syslog socket/dev/log, and then writes them to the appropriate log file.

The word "syslog" stands for several meanings and is often used as one of several names for short:

1. syslog Daemon -a program used to receive, process, and send syslog information. It can remotely send a syslog to a centralized server or write to a local file. Common examples include RSYSLOGD and syslog-ng. In this way of use, people often say "send to syslog".
2. syslog Protocol -A transport protocol that specifies how logs are routed over the network and a definition of the data format for syslog information (see below). It is formally defined in the RFC-5424. For text logs, the standard port is 514, and for encrypted logs, the port is 6514. In this way of use, people often say "transmit via syslog".
3. syslog information -syslog a log message or event in the format that includes a message header with several standard fields. In this way of use, people often say "send syslog".

Syslog information or events include a message header with several standard fields to make analysis and routing easier. They include timestamps, the name of the application, the classification or location of the source of information in the system, and the priority of the event.

The following is a log message containing a syslog message header from the sshd daemon that controls the remote login to the system, which describes a failed login attempt:

<34>1 2003-10-11t22:14:15.003z server1.com sshd--Pam_unix (Sshd:auth): Authentication failure; Logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2

Syslog format and fields

Each syslog message contains an information header with fields that are structured data that makes parsing and routing events easier. Here is the format we used to generate the above Syslog example, and you can match each value to the name of a particular field.

<%pri%>%protocol-version%%timestamp:::d ate-rfc3339%%hostname%%app-name%%procid%%msgid%%msg%n

Below, you'll see some of the syslog fields that are most commonly used when finding or troubleshooting:

Time stamp

The timestamp (the example above is 2003-10-11t22:14:15.003z) implies the time and date of the message being sent in the system. This time may vary when receiving this information on another system. The timestamp in the above example can be decomposed into:

• 2003-10-11 year, month, day.
• T is a required element of the timestamp, which separates the date and time.
• 22:14:15.003 is a 24-hour time, including the number of milliseconds to enter the next second (003).
• Z is an optional element that refers to UTC time, except for Z, this example can also include an offset, for example-08:00, which means that time is offset from UTC by 8 hours, or PST time.

Host Name

The Host Name field (in the example above corresponds to server1.com) refers to the name of the host or the system that sent the message.

Application Name

The Application Name field (in the example above corresponds to Sshd:auth) refers to the name of the program that sent the message.

Priority level

The priority field or abbreviation for the PRI (in the example above) tells us how urgent or severe the event is. It consists of two numeric fields: The device field and the emergency field. The emergency field is from the number 71 that represents the Debug class event until the number 0 that represents the emergency event. The device field describes which process created the event. It is from the number 0 representing the kernel information to the 23 used by the local app.

There are two ways to output a Pri. The first is expressed as a single number, which can be calculated by multiplying the value of the device field by 8, plus the value of the Emergency field: (Device field) (8) + (emergency field). The second is the PRI text, which will be output in the string format "device field. Emergency field". The latter format is easier to read and search, but takes up more storage space.

Linux Log Basics