Log analysis and management in Linux

Log files are helpful in diagnosing and resolving problems in the system, because programs that run on Linux systems often write system messages and error messages to the appropriate log files, so that the system can be "documented" once problems occur. In addition, when the host is attacked, the log file can also help to find traces of the attacker's left.
First, the main log file
In the Linux system, the log data mainly consists of the following three types:
"Kernel and System log"
"User Log"
"Program Log"
The log files of the Linux system itself and most of the server programs are placed in the directory "/var/log" by default . A part of the program common one log file, some programs use a single log file, and some large server programs because of more than one log file, so the "/var/log" directory to establish the appropriate subdirectory to hold the log files, so that the log file directory is clearly structured, You can also quickly locate log files. There is a significant portion of log files that only the root user has permission to read, which guarantees the security of the relevant log information.

Example 1: The list looks at the various log files and subdirectories in the "/var/log" directory.

For some common log files in the Linux system, it is necessary to familiarize themselves with the corresponding uses, so as to find the problem faster and solve various faults in time. The common log files are as follows:

/var/log/messages: logs Linux kernel messages and common log information for various applications, including startup, IO errors, network errors, program errors, and so on. For applications or services that do not use stand-alone log files, it is generally possible to obtain related event logging information from the log file.

/var/log/cron: records The event information generated by the Crond scheduled task.

/VAR/LOG/DMESG: record the various event information of the Linux system during the boot process .

/var/log/maillog: records the e-mail activity that enters or issues the system.

/var/log/lastlog: Recent successful logon events and the last unsuccessful logon event .

/var/log/rpmpkgs: records the list of RPM packages installed in the system.

/var/log/secure: record the event information during user logon authentication .

/var/log/wtmp: Log Each user logon, logoff, and system startup and shutdown events .

/var/run/utmp: records the details of each user who is currently logged on .

Second, log file analysis

The purpose of the analysis log file is to browse the log to find the key information, to debug the system services, to determine the cause of the failure and so on.

For most text-formatted log files, you can view the log content by using text processing tools such as tail, more, less, and cat. For some log files in binary format, you need to use the appropriate query commands.

1. Kernel and System log

The kernel and system log functions are mainly

The default installation of the sysklogd-1.4.1-39.2 package is provided, which installs the KLOGD, syslogd two programs and is controlled via the Syslog service .

Separate messages for the system kernel and messages for various applications. The configuration file used by the Syslog service is "/etc/syslog.conf".

Example 2: View the contents of the Log configuration file "/etc/syslog.conf".

As you can see from the profile "/etc/syslog.conf", log files managed by the SYSLOGD service are the most important log files in the Linux system, and they record the most basic system messages in the Linux system, such as kernel, user authentication, mail, scheduled tasks, and so on. In the Linux kernel, depending on the degree of importance of the log message, it is divided into different priority levels (the smaller the number, the higher the priority, the more important the message).

0 Emerg (Emergency): a condition that causes the host system to become unavailable.

1 alert (warning): a problem that must be taken immediately to resolve.

2 crit (severe): more serious situation.

3 ERR (Error): An error occurred while running.

4 WARNING (reminder): may affect system functions, need to remind users of important events.

5 NOTICE (note): events that do not affect normal functionality, but need attention.

6 Info (info): general information.

For most log files that are unified by the Syslog service, the logging format used is basically the same. The following is an example of the common log file "/var/log/messages" as a simple introduction to the basic format of kernel and system logging.

Example 3: View the last 3 rows of the public log file "/var/log/messages".

Each row in the log file represents a message, and each message consists of a fixed format of four fields.

Time Label: the date and time the message was issued.

Host Name: The name of the computer that generated the message.

Subsystem Name: The name of the application that issued the message.

"Message": the exact content of the message.

In some cases, you can set up a syslog that logs information to a file while sending the log information to the printer for printing, so that no matter how the network intruder modifies the log, it cannot erase the traces of the intrusion . The Syslog Log service is a significant target that is often attacked, destroying it will make it difficult for administrators to uncover traces of intrusions and intrusions, so pay special attention to monitoring their daemons and configuration files.

2. User log

In Wtmp, Utmp, Lastlog and other log files, save the system user login, exit and other related events event message. However, these files are binary data files and cannot be browsed directly using the tail, less and other text viewing tools.

You need to use user query commands such as WHO, W, users, last, and AC to get log information .

"Whocommand "

The Who command is used to query the Utmp file and report each user information that is currently logged on. With this command, the system administrator can audit and process the existing illegal users of the system by looking at them. The WHO default output includes the user name, terminal type, logon date, and remote host.

Example 4: Use the WHO command to view information such as the terminal, login date/time/location for each user who is currently logged in.

"w Command "

used to query the Utmp file and display each user in the current system and the process information it is running.

Example 5: Use the W command to view the user information currently logged in and the processes that are running.

The "Users Command"

The users command prints out the currently logged-on user with a separate line, with each user name displayed with a logon session. If a user has more than one logon session, his user name is displayed with the same number of times.

Example 6: Use the users command to view the current user logon session.

"Last Command"

The last command is used to query the Wtmp file to show all user records that have been logged in since the file was created, and the most recently logged-on user record appears first.

Example 7: Use the last command to view the most recent five user logon records in the system.

"AC Command"

The AC command queries the user login and exit conditions in the Wtmp file , reporting the total time (in hours) that the user is connected. With the user name as a parameter, the "-d" option can be counted on a daily basis. When you do not specify a user name, you can use the "-P" option to separate statistics .

Example 8: Statistics GZJ The user's connection time per day.

Example 9: The total connection time of each user is counted separately.

3. Program Log

In a Linux system, a significant portion of the application does not use the Syslog service to manage the log, but rather the program maintains the log records itself. For example

HTTP Web service program uses two log files Access_log and error_log, generally stored in the "/var/log/httpd/" directory, respectively, recording customer access events, error events;

The FTP service program can record messages related to file uploads and download events in the Xferlog file. Due to the large differences in logging formats for different applications and the fact that the uniform format is not strictly used, this is not described in detail here.

In general, as a qualified system management personnel, should be vigilant, always pay attention to a variety of suspicious situation, regular and random inspection of various system log files, including general information log, network connection log, file transfer log and user log records and so on. When checking these logs, be aware of any unreasonable time or operational records. For example, some of the following unreasonable phenomena.

A, the user in the non-regular time to log on, or the user login system IP address and the past is not the same.

B, log records of user logon failures, especially those that repeatedly attempt to enter the failed log record continuously.

C, illegal use or improper use of superuser privileges.

D. Restart the records of various network services without reason or unlawfully.

E, abnormal log records, such as the incomplete log, or a log file such as wtmp without an unreasonable lack of intermediate record files.

In addition, it is particularly important to remind managers that the logs are not completely reliable, and that smart hackers often clean the scene after they invade the system. Therefore, it is necessary to use the above system commands comprehensively and comprehensively to carry out the examination and inspection, so we should avoid out of context, otherwise we may make wrong judgment.

Third, the Log management policy

In view of the importance of log data, it is necessary to adopt a targeted management strategy to ensure the accuracy, security and authenticity of log data for various log files produced during system operation. In general, the following aspects can be considered.

1. Log Backup and archive

2. Extended Log Retention Period

3. Control Log access rights

4. Centralized management of logs

First, check the operating level of the system

Different runlevel represents the difference between a service or a program that can run with different operating states, and it is clear that the current runlevel will help administrators troubleshoot some application failures. If you are not sure where you are at the current level of operation,

Queries can be made using the runlevel command , which contains the level before the switch and the current level, respectively.

Example 1: View the operating level status of the system, and if the RunLevel has not been previously switched, the first column displays "N".

Second, switching system operating level

When the user needs to convert the system to a different runlevel, it can be done using the init program command , as long as the command parameter is used for the number (0-6) corresponding to the runlevel.

To set the run level that is entered by default when the system is powered on, edit the "/etc/inittab" configuration file.

Example 2: Close the current system.

Example 3: Restart the current system.

Third, set the state of the system service at different operating levels

The Linux system contains a large number of service programs, which are started or terminated by the RC script according to the preset state when switching the runlevel. Of course, it is entirely up to the administrator to decide whether a system service will run at a specific runlevel.

You can use the chkconfig or NTSYSV command when you need to adjust the self-starting state of various system services at different operating levels.

1. Using the NTSYSV Configuration tool

The Ntsysv tool can be run in character mode, providing users with an interactive interface that is designed to centrally configure the startup state of various system services. The NTSYSV command is only used to manage services in the current runlevel, and if the "--level" option is used, the services in the specified run level can be managed. For example, after executing the "NTSYSV--level 35" command, an interactive program interface is opened that can set the startup state of each service at run Level 3 and 5 at the same time.

Use the UP and DOWN ARROW keys to select different service items, press SPACEBAR to toggle the start status, "[*]" means auto start, "[]" means do not start automatically. Press F1 to get a detailed description of a service to determine if it should be loaded.

When you need to set the startup state of a large number of services at the same time, it is more appropriate to choose to use the NTSYSV Configuration tool.

2. Using the Chkconfig Configuration tool

The Chkconfig Configuration tool is similar to NTSYSV, but does not provide an interactive interface, and it is more efficient to choose to use the Chkconfig command when it is necessary to set the startup state of individual services at different operating levels.

use the Chkconfig command with the "--list" option to view the startup status of a particular service at different run levels, and if the service name is not specified, the startup status of all services is displayed.

Example 4: View the startup status of the network service in each run level.

Example 5: View the startup status of all services in the system at each run level.

use the Chkconfig command with the "--level" option to set the startup state of the specified service at the specified run level. The more common command formats are as follows:

Chkconfig--level Run Level list service name On|off

Example 6: Set the network service to off in run Level 2, 4 startup state

"Example 7": Set the Network service startup state in run level 2-5 to On

When the "--level" option is missing, it is sometimes possible to set the state of a specified service at a different run level, but the result of this setting will be affected by the state parameter in the service script file, which is not recommended for use.

Example 8: Use the Chkconfig command without the--level option to set the startup state of the Vncserver service to OFF.

1. Init process

processes in a Linux system are tagged with numbers, and each process's identity token is called a PID.

In the process of booting a Linux system,"/sbin/init" is the first program loaded by the kernel, so the init process will always have a PID number of "1".

After the Init process runs, other programs in the system are executed successively, generating new processes that are called child processes of the INIT process, whereas the Init process is the parent process of those processes. Of course, these sub-processes can also further generate their own sub-processes, in turn continue to multiply, and eventually constitute a leafy process tree, together to provide users with services.

The init process is the "ancestor" of all processes that keep the entire Linux system running, so the init process is not allowed to be easily terminated. When you need to switch different system operating states, you can assign the correct execution parameters to the INIT program, which is done by Init itself.

2. Inittab configuration file

The inittab file is located in the "/etc" directory, which is the configuration file for the INIT program , and after the Init process runs, it starts the scripts and programs that need to be run in the system, starting with what is configured in the file.

To view the valid configuration lines in the "/etc/inittab" file:

In the "/etc/inittab" file, each line is treated as a valid configuration record in addition to the comment information and the blank line that begin with the "#" sign. The four fields in each configuration record are split using a half-width colon ":" with the following basic format:

Id:runlevels:action:process

The meaning is as follows:

Tag: Run level: Action type: Program or script

The functions of each field are as follows:

"ID---tag field "

A marker field can be made up of 1-4 characters to distinguish it from the other lines of the configuration. The ID tag field for each configuration record in the Inittab file should be unique.

"Runlevels---run Level field "

In the Linux system, various system services are combined to form different collocation relationships, respectively, to meet different system requirements, the system at run time, the use of each service collocation called "RunLevel." The default system RunLevel is seven, and its functions and services are the same.

0: Shutdown state, the host will be turned off when this level is used.

1: Single user mode, no password authentication is required to log into the system, more for system maintenance.

2: Multi-user mode for character interface (network not supported).

3: The full multi-user mode of the character interface, most server hosts run at this level.

4: Not assigned to use.

5: Graphical interface of multi-user mode, provides a graphical desktop operating environment.

6: Reboot, the host will be restarted when this level is used.

Action ---action Type field

The Action Type field describes the action category for the row configuration, which is generally fixed, and the more commonly used types are as follows:

Initdefault: Sets the run level that is entered by default when the system is initialized.

sysinit: Sets the operation script for system initialization.

wait: The init process waits for the script operation for the row configuration to complete before proceeding with other operations.

Ctrlaltdel: Sets the action after the user presses the Ctrl+alt+delete key combination.

powerokwait: Sets what action to take when the system receives a power-back signal.

respawn: Restart the process once the corresponding process for the line configuration is terminated.

"Process---program or script field "

This field is used to specify the actual operation of the line configuration, which can be a specific command, a script, and so on.

3. rc.sysinit script File

Rc.sysinit is the system initialization script called by the Init process , located in "/etc/rc.d/rc.sysinit", which is also set in the Inittab configuration file.

The main completion of the Rc.sysinit script includes setting up the network, hostname, loading the file system, setting up the clock and a series of initialization work .

4. rc Script File

The RC script is also called by the Init process, and the file is in "/ETC/RC.D/RC".

The RC script loads and terminates different system services by specifying different level parameters to enter the appropriate system runlevel.

In the "/ETC/RC.D/RC0.D"---"/etc/rc.d/rc6.d" directory, some special symbolic link files are saved.

The RC script initiates or terminates the operation of the associated service program based on the link file names in these directories and the system service scripts that they are linked to.

The linked file in the "/ETC/RC.D/RCX.D" directory has a common law: The file name begins with K or S, the middle is the number ordinal, and finally the service script name in the system, and the original service script file that is linked is located in the "/ETC/RC.D/INIT.D" directory .

Where the file beginning with S represents the start of the corresponding service , a file beginning with k indicates terminating the corresponding service , and the middle number indicates the order of execution when the service is started or terminated.

The various system service scripts located in the "/ETC/RC.D/INIT.D" directory can basically be executed directly from a script file or managed through the Service Control tool, for example, adding "Start", "Stop", "restart", "status" and other parameters such as the start, termination, restart, query status and other operations of the corresponding service. For system service programs, it is generally recommended to start or terminate this way instead of using the kill process command. For server programs in the actual working environment, it is recommended to use the "reload" parameter to reload the configuration instead of rigidly using "restart" to restart the service so as not to cause client access disruption and unnecessary loss.

5. rc.local script File

The rc.local script is an additional startup control file located in "/etc/rc.d/rc.local". rc.local scripts are typically loaded by the RC script at the end , and are primarily designed to provide a way for administrators to set their own startup commands. Some command actions that need to be performed automatically after a reboot can be added to the file.

Log analysis and management in Linux