Analysis and troubleshooting of Linux system failure--log analysis

When dealing with various failures in the Linux system, the symptoms of the failure are the first to be discovered, and the cause of the failure is the key to eventual troubleshooting. Familiar with the Linux system log management, understand the common fault analysis and solution, will help administrators quickly locate the point of failure, "the remedy" to solve various system problems in a timely manner.

The main log files consist of the following three types:
A. Kernel and system logs: This log data is managed by the system service Syslog, which determines where the kernel messages and various system program messages are logged, based on the settings in the master configuration file "/etc/syslog.conf". There are quite a few programs in the system that send their own log files to syslog management, so the log records used by these programs also have similar formats.
B. User log: This log data is used to record information about the Linux system user login and exit system, including user name, login terminal, logon time, source host, process operation in use, etc.
C. Program log: Some applications will choose their own to manage a log file independently (rather than to the Syslog service management), used to record the program during the operation of various event information. Because these programs are only responsible for managing their own log files, there may be significant differences in the logging format used by different programs.

#列表查看 the various log files and subdirectories in the "/var/log" directory.

for some common log files in the Linux system, it is necessary to familiarize themselves with the corresponding uses, so as to find the problem faster and solve various faults in time. such as:
1. >/var/log/messages: Records the Linux kernel messages and common log information for various applications, including startup, IO errors, network errors, program failures, and so on. For applications or services that do not use stand-alone log files, it is generally possible to obtain related event logging information from the file.
2. >/var/log/cron: Log event messages generated by Crond scheduled tasks.
3. >/VARLOG/DMESG: Record the various event information of the Linux system during the boot process.
4. >/var/log/maillog: Record the e-mail activity that enters or issues the system.
5. >/var/log/lastlog: Recent successful logon events and last unsuccessful logon events.
6. >/var/log/rpmpkgs: Install each RPM package list information in the recording system.
7. >/var/log/secure: Record The event information in the user login authentication process.
8. >/var/log/wtmp: Log Each user logon, logoff, and system startup and shutdown events.
9. >/var/log/utmp: Record details of each user currently logged in

Second, log file analysis
Familiar with the main log in the system, we will be on the log file analysis method to do understanding. The purpose of the analysis log file is to find the key information through the browsing log, debug the system service, determine the cause of the failure and so on. Here, the basic format of the three types of log files and analysis methods.
For most text-formatted log formats (such as kernel and system logs, most program logs), you can view the log content using text processing tools such as tail, more, less, and cat. For some binary format log files (eg: User logs), you need to use the appropriate query commands.
1. Kernel and System log:
The kernel and syslog features are mainly provided by the default installed syslogd-1.4.1-39.2 package, which installs KLOGD, syslogd two programs, and is controlled by the Syslog service, which is used to record the messages of the system kernel and the messages of various applications, respectively. The configuration file used by the Syslog service is "/etc/syslog.conf".
Typically, the kernel and most system messages are recorded in the public log file "/var/log/messages", while other program messages are recorded in different files, and log messages can be logged to a specific storage device or sent directly to the user.
# # #查看日志配置文件 content in '/etc/syslog.conf '

2. User log
in Wtmp, Utmp, Lastlog and other log files, save the system user login, exit and other related events event message. However, these files are binary data files, can not directly use the tail, less and other text viewing tool process browsing, you need to use the WHO, W, users, Last and AC and other user query commands to obtain the log information.
This is no longer a demonstration.
3. Program Log
in a Linux system, there are a significant portion of applications that do not use syslog services to manage logs. Instead, the program maintains its own log records. For example, the HTTPD Web Services program uses two log files Access_log and error_log, typically stored in the "/var/log/httpd" directory, recording customer access events, error events, and the FTP service program can be uploaded with the file, Download event-related messages are logged in the Xferlog file. Due to the different application logging format is large, and there is no strict use of uniform format, here is an unknown solution!
Exception: Server log Distribution Management policy:
In view of the importance of log data, it is necessary to use a targeted management strategy to ensure the accuracy, security and authenticity of log data for various log files generated during the system operation. In general, the following aspects can be considered.
log backup and archive: Log files are also important data and need to be backed up and archived.
: Extended Log retention period: Log data should be kept as long as possible in the event of a rich storage space.
: Control log access: The log data may contain various kinds of sensitive information, such as: Account number, password, etc. Therefore, it is necessary to strictly control its access rights.
Centralized Management log: Use a centralized log server to manage log records sent by each server. The advantage is to facilitate the collection, collation and analysis of the log, to eliminate accidental loss, malicious tampering or deletion.
eg: Server A (IP address is 173.17.17.3/24), which is used to centrally save the log records.
The log records generated by the Crond service in client B (IP address 173.17.17.11/24) are uniformly saved to the "/var/log/cron" file in Server A.
set up log Server a
in log Server A, you need to edit the startup parameter profile "/etc/sysconfig/syslog" of the Syslog log service to change the contents of the syslogd_options variable to "-r-x-M 0". Where the "-r" option means that log records sent by other hosts are allowed, the "-X" option means no process DNS domain name resolution, "-M" indicates the time stamp interval for logging (set to 0 to disable the feature), which can be obtained by viewing the Man manual page of the SYSLOGD program
*: Modify the "/etc/sysconfig/syslog" file of log Server A, add the central management configuration parameter "-r" and restart the Syslog service.
1. Vi/etc/sysconfig/syslog//Modify Syslogd_options line syslogd_options= "-r-x-M 0"
2. Service syslog Restart

Set Client B
in client B, you need to modify the "/etc/syslog.conf" configuration file to write log messages for Cron scheduled tasks to the "/var/log/cron" file in Server A. Specifies the "@173.17.17.3" format to use when specifying the host address to write to the log.
*: Modify the "/etc/syslog.conf" file of client B, locate the configuration line of the cron log, change the log send location to "@173.17.17.3", and restart the Syslog service.
1. vi/etc/syslog.conf
2. cron.* @173.17.17.3
3. Service syslog Restart

Verifying log Centralized management features
execute the "crontab-e" command in client B, write a scheduled task information and save the exit, then view the "/var/log/cron" log file in this computer, and you will find that there are no new records.

Analysis and troubleshooting of Linux system failure--log analysis