Linux System log Analysis

The Linux system has a very flexible and powerful logging function, which can save almost all the operation records and retrieve the information we need.

The default log daemon for most Linux distributions is Syslog, which is located in/etc/syslog or/etc/syslogd or/ETC/RSYSLOG.D, and the default configuration file is/etc/syslog.conf or rsyslog.conf, any program that wants to generate a log can send information to the syslog.

Linux system cores and many programs generate various error messages, warning messages, and other prompts that are useful for administrators to understand the state of the system, so they should be written to a log file.

The procedure to complete this process is syslog. Syslog can save logs to different files based on the category and priority of the log.

For example, for easy reference, you can separate the kernel information from other information and save it to a separate log file. In the default configuration, log files are usually saved in the "/var/log" directory.

Log type

The following are common log types, but not all Linux distributions contain these types:

type	Description
Auth	Logs generated when the user authenticates, such as the login command, the SU command.
Authpriv	Similar to auth, but can only be viewed by specific users.
Console	Messages for the system console.
Cron	The log that is generated when the system periodically executes scheduled tasks.
Daemon	Logs generated by some daemons.
Ftp	FTP Service.
Kern	System kernel messages.
Local0.local7	Used by custom programs.
Lpr	Related to printer activity.
Mail	Mail logs.
Mark	Generates a timestamp. The system outputs the current time to the log file at intervals, and each line is formatted like May 11:17:09 RS2-MARK-to infer the approximate time the system has failed.
News	The message generated by the Network News Transfer Protocol (NNTP).
Ntp	The message generated by the Network Time Protocol (NTP).
User	The user process.
Uucp	UUCP subsystem.

Log priority

Common log priorities See subscript:

Priority Level	Description
Emerg	In an emergency, the system is not available (such as a system crash) and is generally notified to all users.
Alert	Requires immediate repair, such as system database corruption.
Crit	Dangerous situations, such as hard drive errors, can hinder some of the program's functionality.
Err	Generic error message.
Warning	Warning.
Notice	Not an error, but may need to be handled.
Info	Generic messages are generally used to provide useful information.
Debug	The information that is generated by the debug program.
None	No priority, no log messages are logged.

Common Log files

The system log is managed by a service called syslog, such as the following log files are driven by the Syslog log service:

/var/log/boot.log: Recorded the system in the boot process events, is the Linux system post process display information

/var/log/lastlog: Record the last time the user successfully logged in, login IP and other information

/var/log/messages: Documenting common system and service error messages for Linux operating systems

/var/log/secure:linux system security log, record user and workgroup deterioration situation, user login authentication situation

/VAR/LOG/BTMP: Logs the user, time, and remote IP address of a Linux login failure

/var/log/syslog: Log only warning messages, often information about system problems, and use Lastlog to view

/var/log/wtmp: This log file permanently records each user logon, logoff and system startup, shutdown events, using the last command to view

/var/run/utmp: The log file records information about each user who is currently logged on. such as WHO, W, users, finger and so on need to access this file

/var/log/syslog or/var/log/messages stores all global system activity data, including boot information. Debian-based systems such as Ubuntu store them in/var/log/syslog, while RedHat-based systems such as RHEL or CentOS store them in/var/log/messages.
/var/log/auth.log or/var/log/secure stores logs from pluggable authentication modules (PAM) , including successful logins, failed login attempts, and authentication methods. Ubuntu and Debian store authentication information in/var/log/auth.log, while RedHat and CentOS store that information in/var/log/secure.

Detailed description of the log file:

/var/log/boot.log

This file records the events that occurred during the boot process, which is the information displayed by the Linux system post process, as shown in 1:

/var/log/syslog

The default Centos,fedora does not generate the log file, but you can configure/etc/syslog.conf to have the log file generated by the system.

It is different from the/etc/log/messages log file, it only records the warning message, often is the system problem information, so should be concerned about the file.

To have the log file generated by the system, add: *.warning/var/log/syslog The log file can record information such as error password, sendmail problem, su command execution failure when the user logged in/etc/syslog.conf.

This log file records the most recent successful logon event and the last unsuccessful logon event that was generated by login. Each time a user logs on, the file is a binary file and needs to be viewed using the Lastlog command, which displays the login name, port number, and last logon time based on the UID. If a user has never logged in, it is displayed as "**never logged in**". This command can only be performed with root privileges. Simply enter the Lastlog command and you will see a message similar to Figure 4:

1234567891011121314151617181920212223

[[email protected] etc]# lastlogUsername         Port     From             Latestroot             pts/0    110.87.109.232   Wed Nov  2 10:34:20 +0800 2016bin                                        **Never logged in**daemon                                     **Never logged in**adm                                        **Never logged in**lp                                         **Never logged in**sync                                       **Never logged in**shutdown                                   **Never logged in**halt                                       **Never logged in**mail                                       **Never logged in**uucp                                       **Never logged in**operator**Never logged in**games                                      **Never logged in**gopher                                     **Never logged in**ftp                                        **Never logged in**nobody                                     **Never logged in**vcsa                                       **Never logged in**saslauth                                   **Never logged in**postfix                                    **Never logged in**sshd                                       **Never logged in**ntp                                        **Never logged in**suda             pts/2    222.79.79.120    Tue Oct 25 16:00:01 +0800 2016

/var/log/wtmp

This log file permanently records each user's logon, logoff, and system startup, downtime events. Therefore, as the system uptime increases, the size of the file becomes larger and higher, depending on the number of times the system user logs on. The log file can be used to view the user's login record, the last command to access this file to obtain this information, and in reverse order from back to front to display the user's log-in record, can also be based on user, terminal TTY or time display corresponding records.

/var/run/utmp

The log file records information about each user who is currently logged on. As a result, the file changes as the user logs on and off the system, preserving only the user records that were online at that time and not keeping permanent records for the user. The system needs to query the current user state of the program, such as WHO, W, users, finger and so on need to access this file. The log file does not include all the exact information, because some burst errors terminate the user logon session, and the system does not update the UTMP record in a timely manner, so the log file record is not entirely trustworthy.

The 3 files mentioned above (/var/log/wtmp,/var/run/utmp,/var/log/lastlog) are the key files of the log subsystem, all of which record the user login. All records for these files contain timestamps. These files are stored in binary, so they cannot be viewed directly with commands such as less and cat, but they need to be viewed through these files using related commands. Where the data structures of utmp and wtmp files are the same, and Lastlog files use additional data structures, the concrete data structures for them can be queried using the man command.

Each time a user logs in, the login program looks at the UID of the user in the file lastlog. If present, the user's last logon, logoff time, and hostname are written to standard output, then the login program logs the new logon time in Lastlog, opens the Utmp file and inserts the user's utmp record. This record is always deleted when the user logs on and exits. utmp files are used by various commands, including WHO, W, users, and finger.

Next, the login program opens the file wtmp additional user's utmp record. When a user logs on and exits, the same utmp record with the update timestamp is appended to the file. The Wtmp file is used by the last program.

Linux System log Analysis