Basic tutorials for Linux system log analysis

First, we'll describe the basics of what Linux logs are, where to find them, and how they are created

Linux System Log

Many valuable log files are created automatically for you by Linux. You can find them in the/var/log directory. Here's what this directory looks like in a typical Ubuntu system:

Some of the most important Linux system logs include:

/var/log/syslog or/var/log/messages stores all the global system activity data, including power-on information. Debian-based systems such as Ubuntu store them in/var/log/syslog, while RedHat based systems such as RHEL or CentOS store them in/var/log/messages.

/var/log/auth.log or/var/log/secure stores logs from pluggable authentication modules (PAM), including successful logins, failed logon attempts, and authentication methods. Ubuntu and Debian store authentication information in/var/log/auth.log, while RedHat and CentOS store that information in/var/log/secure.

/var/log/kern stores kernel error and warning data, which is especially useful for troubleshooting custom kernel-related failures.

/var/log/cron stores information about cron jobs. Use this data to make sure your cron job is running successfully.

Digital Ocean has a complete tutorial on these files, which describes how Rsyslog creates them in common distributions such as RedHat and CentOS.

The application will also write to the log file in this directory. For example, common server programs like Apache,nginx,mysql can write log files in this directory. Some of these log files are created by the application itself, others through syslog (see below).

What is Syslog?

How is the Linux system log file created? The answer is through the syslog daemon, which listens for log information on the syslog socket/dev/log and writes them to the appropriate log file.

The word "syslog" stands for several meanings and is often used as one of several names as follows:

Syslog Daemon-A program used to receive, process, and send syslog information. It can remotely send syslog to a centralized server or write to a local file. Common examples include RSYSLOGD and syslog-ng. In this way of use, people often say "send to syslog".

Syslog Protocol-A transport protocol that specifies how logs are transmitted over the network and a definition of the data format for syslog information (see below). It is formally defined in the RFC-5424. For a text log, the standard port is 514, and for the encrypted log, the port is 6514. In this way of use, people often say "through the syslog transmission."

Syslog information-syslog The format of log information or events, which includes a message header with several standard fields. In this way of use, people often say "send syslog".

The Syslog information or event includes a message header with several standard fields to make parsing and routing easier. They include the timestamp, the name of the application, the classification or location of information sources in the system, and the priority of the event.

The following is a log message containing the syslog header, which is from the sshd daemon, which controls remote logins to the system, which describes a failed logon attempt:

<34>1 2003-10-11t22:14:15.003z server1.com sshd-Pam_unix (Sshd:auth): Authentication failure; Logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2

Syslog format and fields

Each syslog information contains a header with a field that is structured so that it is easier to parse and route events. Here is the format we used to generate the above Syslog example, where you can match each value to a specific field name.

Copy Code

The code is as follows:

<%pri%>%protocol-version%%timestamp:::d ate-rfc3339%%hostname%%app-name%%procid%%msgid%%msg%n

Below, you will see some of the most commonly used syslog fields in finding or scheduling:

Time stamp

The timestamp (the above example is 2003-10-11t22:14:15.003z) implies the time and date that the information was sent in the system. This time may be different when you receive this information on another system. The timestamp in the example above can be decomposed into:

2003-10-11 year, month, day.

T is a required element of the timestamp, separating the date and time.

22:14:15.003 is a 24-hour time, including the number of milliseconds to enter the next second (003).

Z is an optional element that refers to the UTC time, except for Z, which can include an offset, such as 08:00, which means that the time is offset from UTC for 8 hours, or PST time.

Host Name

The Host Name field (corresponding to server1.com in the example above) refers to the name of the host or the system that sends the information.

Application Name

Applying a Name field (corresponding to Sshd:auth in the example above) refers to the name of the program that sent the message.

Priority level

The priority field or the abbreviation for the PRI (corresponding in the above example) tells us how urgent or severe the event is. It consists of two numeric fields: Device fields and Emergency fields. The emergency field is from the number 71 representing the Debug class event until the number 0 represents the emergency. The device field describes which process created the event. It is from the number 0 that represents the kernel information to the 23 that is used for the local application.

The Pri has two modes of output. The first is represented as a separate number, which can be calculated by multiplying the value of the device field by 8, plus the value of the Emergency field: (Device field) (8) + (emergency field). The second is the PRI text, which is output in the string format of "device field. Emergency field." The latter format is easier to read and search, but takes up more storage space.

Analyzing Linux Logs

There is a lot of information in the log that you need to deal with, although sometimes it's not easy to imagine. In this article we'll look at some examples of basic log analysis You can do now (search only). We will also involve some more advanced analysis, but these require your early efforts to make the appropriate settings and later save a lot of time. Examples of advanced analysis of data include generating rollup counts, filtering valid values, and so on.

We'll start by showing you how to use several different tools on the command line, and then show how a log management tool can automate most of the heavy work and make log analysis easier.

Using Grep Search

Searching for text is the most basic way to find information. The most common tool for searching text is grep. This command-line tool is available in most Linux distributions and allows you to search for logs with regular expressions. A regular expression is a pattern written in a special language that recognizes matching text. The simplest pattern is to enclose the string you want to find in quotes.

Regular expressions

This is an example of the "User Hoover" in the authentication log of the Ubuntu system:

Copy Code

The code is as follows:

$ grep "User Hoover"/var/log/auth.log

Accepted password for Hoover from 10.0.2.2 Port 4792 ssh2

Pam_unix (Sshd:session): Session opened to User Hoover by (uid=0)

Pam_unix (Sshd:session): Session closed for user Hoover

It can be difficult to build an exact regular expression. For example, if we want to search for a number similar to the port "4792", it may also match the timestamp, URL, and other unwanted data. The following example in Ubuntu, which matches an Apache log that we don't want.

Copy Code

The code is as follows:

$ grep "4792"/var/log/auth.log

Accepted password for Hoover from 10.0.2.2 Port 4792 ssh2

74.91.21.46--[31/mar/2015:19:44:32 +0000] "get/scripts/samples/search?q=4972 http/1.0" 404 545 "-" "-"

Surround Search

Another useful tip is that you can use grep to do a surround search. This will show you what a match is before or after a few lines. It can help you debug things that cause errors or problems. The B option shows the previous lines, the A option shows the following lines. For example, we know that when a person fails to log on as an administrator, and their IP does not have a reverse resolution, it means they may not have a valid domain name. This is very suspicious!

Copy Code

The code is as follows:

$ grep-b 3-a 2 ' Invalid user '/var/log/auth.log

APR 17:06:20 ip-172-31-11-241 sshd[12545]: Reverse mapping checking getaddrinfo for 216-19-2-8.commspeed.net [216.19.2 .8] Failed-possible break-in attempt!

APR 17:06:20 ip-172-31-11-241 sshd[12545]: Received disconnect from 216.19.2.8:11:bye Bye [PreAuth]

APR 17:06:20 ip-172-31-11-241 sshd[12547]: Invalid user admin from 216.19.2.8

APR 17:06:20 ip-172-31-11-241 sshd[12547]: input_userauth_request:invalid user admin [PreAuth]

APR 17:06:20 ip-172-31-11-241 sshd[12547]: Received disconnect from 216.19.2.8:11:bye Bye [PreAuth]

Tail

You can also use grep and tail to get the last few lines of a file, or to track logs and print them in real time. This is useful when you make interactive changes, such as starting a server or testing code changes.

Copy Code

The code is as follows:

$ tail-f/var/log/auth.log | grep ' Invalid user '

APR 19:49:48 ip-172-31-11-241 sshd[6512]: Invalid user ubnt from 219.140.64.136

APR 19:49:49 ip-172-31-11-241 sshd[6514]: Invalid user admin from 219.140.64.136

A detailed description of grep and regular expressions is outside the scope of this guide, but Ryan's tutorials has a more in-depth description.

The log management system has higher performance and more powerful search capabilities. They usually index the data and make parallel queries, so you can quickly search for gigabytes or terabytes of logs in seconds. In contrast, grep takes several minutes, and in extreme cases may even be hours. The log management system also uses a query language similar to Lucene, which provides a simpler syntax for retrieving numbers, fields, and more.

Parsing with cut, AWK, and Grok

Linux provides multiple command-line tools for text parsing and parsing. It is useful when you want to quickly parse small amounts of data, but it can take a long time to process large amounts of data.

Cut-and

The Cut command allows you to parse a field from a delimited log. A separator is an equal sign or comma that separates a field or key value pair.

Suppose we want to parse out the user from the following log:

Copy Code

The code is as follows:

Pam_unix (Su:auth): Authentication failure; Logname=hoover uid=1000 euid=0 tty=/dev/pts/0 ruser=hoover rhost=

We can use the cut command to get the text of the eighth field, separated by an equal sign, as follows. This is an example of an Ubuntu system:

Copy Code

The code is as follows:

$ grep "Authentication Failure"/var/log/auth.log | cut-d ' = '-F 8

Root

Hoover

Root

Nagios

Nagios

Awk

In addition, you can also use awk, which provides a more powerful analytic field function. It provides a scripting language, and you can filter out almost anything irrelevant.

For example, suppose we have the following line of logs in the Ubuntu system, and we want to extract the name of the user who failed the login:

Copy Code

The code is as follows:

Mar 08:28:18 ip-172-31-11-241 sshd[32701]: input_userauth_request:invalid user Guest [PreAuth]

You can use the awk command as follows. First, a regular expression/sshd.*invalid user/is used to match the sshd invalid user row. Then use {print $} to print the Nineth field based on the default separator space. This will output the user name.

Copy Code

The code is as follows:

$ Awk '/sshd.*invalid user/{print $} '/var/log/auth.log

Guest

Admin

Info

Test

Ubnt

You can read more about how to use regular expressions and output fields in the Awk User's Guide.

Log Management System

The log management system makes parsing easier, allowing users to quickly analyze a lot of log files. They can automatically parse standard log formats, such as common Linux logs and Web server logs. This can save a lot of time because you don't have to think about writing parsing logic when dealing with system problems.

The following is an example of a sshd log message that resolves each remotehost and user. This is a screenshot of loggly, which is a cloud-based log management service.

You can also customize parsing for non-standard formats. A common tool is Grok, which uses a common regular expression library to parse the original text into structured JSON. The following is a case configuration in which Grok resolves kernel log files in Logstash:

Copy Code

The code is as follows:

filter{

Grok {

Match => {"message" => "%{ciscotimestamp:timestamp}%{host:host}%{word:program}%{notspace}%{notspace}%{number :d uration}%{notspace}%{greedydata:kernel_logs} "

}

}

The following figure is the result of the output after Grok parsing:

Filter with Rsyslog and AWK

Filtering allows you to retrieve a specific field value instead of Full-text search. This makes your log analysis more accurate because it ignores matches that are not needed from other parts of the log information. To search for a field value, you first need to parse the log or at least have a way to retrieve the event structure.

How to filter an application

Typically, you may want to read only one application log. If your application saves records to a single file, it will be easy. It's more complicated if you need to filter an application in a clustered or centralized log. Here are a few ways to implement:

Parse and filter logs with the Rsyslog daemon. The following example writes the log applied by sshd to a file named Sshd-message, and then discards the event so that it does not recur elsewhere. You can add it to your rsyslog.conf file to test this example.

Copy Code

The code is as follows:

:p Rogramname, IsEqual, "sshd"/var/log/sshd-messages

&~

Extract the value of a particular field, such as sshd user name, with a command-line tool like awk. Here is an example of an Ubuntu system.

Copy Code

The code is as follows:

$ Awk '/sshd.*invalid user/{print $} '/var/log/auth.log

Guest

Admin

Info

Test

Ubnt

Use the log management system to automatically parse the log, and then click Filter on the application name that you want. The following is a screenshot of the syslog domain that was extracted from the loggly Log Management service. We filter the application name "Sshd", as shown in the Venn diagram icon.

How to filter errors

A person most want to see errors in the log. Unfortunately, the default syslog configuration does not directly export the severity of the error, making it difficult to filter them.

Here are two ways to solve the problem. First, you can modify your rsyslog configuration to output the severity of the error in the log file, making it easy to view and retrieve. In your rsyslog configuration you can add a template with Pri-text, as follows:

Copy Code

The code is as follows:

"<%pri-text%>:%timegenerated%,%hostname%,%syslogtag%,%msg%n"

This example will be output in the following format. You can see the ERR in this message indicating the error.

Copy Code

The code is as follows:

　　 : Mar 18:18:00,hoover-virtualbox,su[5026]: pam_authenticate:authentication failure

You can retrieve the error message using AWK or grep. In Ubuntu, for this example, we can use some grammatical features, such as. And, they will only match this field.

Copy Code

The code is as follows:

$ grep ' .err> '/var/log/auth.log

　　 : Mar 18:18:00,hoover-virtualbox,su[5026]: pam_authenticate:authentication failure

Your second option is to use the log management system. A good log management system can automatically parse Syslog messages and extract error fields. They also allow you to filter specific errors in log messages with a simple click.

The following is a screenshot of loggly that shows the syslog domain highlighting the severity of the error, indicating that we are filtering the error: