RHEL7.0 Log System

RHEL7.0 Log System

This article is a note I took when studying RHCE7.0. I hope it will be helpful to you.

System log Architecture

System Logs

The process and the operating system kernel must be able to log the time before occurrence. These logs can be used for System Review and troubleshooting. By convention, these logs are permanently stored in the/var/log directory.

Log System in RHEL7
Red Hat Enterprise Linux has a built-in standard logging system based on the System Log protocol. Many programs use this system to record events and organize them into log files. System log messages in Red Hat Enterprise Linux 7 are processed by two services: systemd-journald and rsyslog.

The systemd-journald daemon provides an improved log management service that collects logs from the kernel, early stages of the startup process, standard output, and system logs, and error messages during startup and running of the daemon. It writes these messages to a structured event log, which is not retained between restarts by default. This allows the system log messages and time missed by system logs to be collected to a central database. System messages can be forwarded to rsyslog by systemd-journald for further processing.
The rsyslog service then arranges system log messages by type (or device) and priority, and writes them to permanent files in the/var/log directory.

The/var/log directory stores various system and service-specific log files maintained by rsyslog.

Overview of System Log Files
Log File Usage
/Var/log/messages most system log messages are recorded here. The exception is identity authentication, email processing-related messages for regularly running jobs, and purely debugging-related information.
/Var/log/secure security and authentication-related messages and error log files.
/Var/log/maillog the log files related to the email server.
/Var/log/boot. log messages related to system startup are recorded here.

System log files
Many programs use the syslog protocol to record the time to the system. Each log message is classified by device (Message type) and priority (Message severity. Available devices are outlined on the rsyslog. conf (5) man page.

The eight priorities are standardized and rated as described below
Encoding priority severity
0 emerg system unavailable.
1 alert must take immediate action.
2. Severe crit condition.
3. err is very serious.
4. warning status.
5. notice normal but important events.
6. info events.
7. debug-level messages.

The rsyslogd Service uses the device and priority of log messages to determine how to process them. This is done through the/etc/rsyslog. conf file and the *. conf file in/etc/rsyslog. d. The program and administrator can put a custom file with the. conf suffix into the/etc/rsyslog. d directory to change the rsyslogd configuration without being overwritten by rsyslog updates.

In/etc/rsyslog. conf, ##### RULES ##### some packages define commands for saving log messages. The left side of each line indicates the device and severity of the log message that matches the command. The device and severity fields in the rsyslog. conf file may contain * characters as wildcards, representing all devices and all severity levels. The right of each line indicates the file to save the log message. Log messages are usually stored in files in the/var/log directory.

Log files are maintained by the rsyslog service. The/var/log directory contains various log files specific to certain services.
For example, the Apache Web server or Samba writes its own log files to the subdirectory corresponding to the/var/log directory.

Messages processed by rsyslog may appear in multiple different log files. To avoid this problem, set the Severity field to none,
Indicates that all messages directed to this device are not added to the specified log file.

In addition to recording the log file messages to the Chinese and foreign countries, you can also print them to all the terminals that have logged on to the user. In the default rsyslog. conf file, perform this operation on all messages with the priority of "emerg.

Log File Rotation
Logs are rotated using the logrotate utility to prevent them from filling up the file systems that contain/var/log. When a log file is rotated, It is renamed using the name extension. The name extension indicates the rotation date. If the file is rotated after January 1, October 30, 2014, the original/var/log/messages file is changed to/var/log/messages-20141030. After turning the original file, a new log file will be created and a service will be notified to perform the write operation on it.

After rotation several times (usually four times), discard the original log file to release disk space. The cron job runs the logrotate program once a day to check whether any logs need to be rotated. Most log files rotate once a week, but the logrotate rotation speed is sometimes faster, Sometimes slower, or rotated when the file reaches a specific size.

For details about logrotate configuration, see logrotate (8) man page.

Analyze System Log entries
The system log file written by rsyslog displays the oldest message at the beginning of the file and the latest message at the end of the file. All log entries in the log files managed by rsyslog are recorded in the standard format. The following example describes the log messages in the/var/log/secure log file.

① Feb 11 20:11:48 ② localhost ③ sshd [1438] ④ Failed password for student from 172.25.0.10 port 59344

① Record the timestamp of the log entry
② Host sending the log message
③ Program or process that sends the log message
④ Actual message sent

Use tail to monitor log files
Monitor one or more log files of an event, which is particularly helpful for reproduction. The tail-f/path/to/file command outputs the last 0 lines of the specified file, and continues to output them in the newly written to the monitored file.

Use logger to send system log messages
The logger command can send messages to the rsyslog service. By default, messages whose severity is notice (user. notice) are sent to the device user, unless an additional message is specified using the-p option. The test is particularly useful for rsyslog configuration changes.

To send messages to rsyslogd and record them in the/var/log/boot. log File, run the following command:
Logger-p local7.notice LOGSTRING

View systemd log entries

Search for events using journalctl
Systemd logs store log data in structured binary files with indexes. This data contains additional information related to log events. For example, for system log time, this can contain the device and priority of the original message.

In Red Hat Enterprise Linux 7, the content of the systemd log is stored in/run/log by default and cleared after restart. This setting can be changed by the system administrator.

The journalctl command displays complete system logs from the oldest log entries.

The journalctl command highlights the information with a priority of notice or warning in rough text, and highlights messages with an error or higher level in red text.

The key to the successful use of logs for troubleshooting and review is to limit the log search to only display the relevant output.

By default, journalctl-n displays the last 10 log entries. You can specify the last number of log entries by using optional parameters. To display the last five log entries, run
Journalctl-n 5

When troubleshooting a problem, the log file output is filtered out based on the priority of the log entry. There is a unique journalctl-p parameter that can take names or numbers of known priorities as parameters, displays all entries at the specified level and higher level.
Journalctl-p err

Similar to the tail-f command, journalctl-f outputs the last 10 lines of logs and writes New log entries to the logs to continue outputting them.
Journalctl-f

It is very useful to limit the output to a specific time period when you look up events of a specific time period. The journalctl command has two options to limit the output to a specific time range, the -- since and -- until options, respectively, both accept time parameters in the format of YYYY-MM-DD hh: mm: ss. If the date is omitted, the command assumes that the log is the current day. If the time part is omitted, it is assumed that the log is a whole day from 00:00:00, except for the date and time fields, the two options also accept yesterday, today, and tomorrow as the valid date parameter.

Output all log entries recorded on the current day.
Journalctl -- since today

Output log entries from 20:30:00, January 1, February 10, 2014 to 12:00:00, January 1, February 13, 2014.
Journalctl -- since "20:30:00" -- until "12:00:00"

In addition to visible log Content, log entries also contain fields that can be viewed only when detailed output is enabled. All additional fields displayed can be used to filter log query output. This can be used to reduce the output of complex searches for specific events in search logs.
Journalctl-o verbose

Other options for searching rows for specific processes or events include
_ COMM command name
_ Path of the executable file of the EXE process
_ PID of the PID Process
_ UID: UID of the user who runs the process
_ SYSTEMD_UNIT: The systemd unit for starting the process.

You can combine multiple options. For example, the query displays all log entries related to the process where the sshd. service file is started and the PID is 1182.
Journalctl _ SYSTEMD_UNIT = sshd. service _ PID = 1182

Save the systemd log

Permanent storage of system logs
By default, the systemd log is saved in/run/log/journal, which means it will be cleared when the system restarts. This log is a new mechanism in Red Hat Enterprise Linux 7, and for most installations, detailed logs from the last startup are sufficient.

If the/var/log/journal directory exists, the log is recorded in this directory. This feature enables you to use historical data immediately after startup. However, not all data is permanently retained even in permanent logs. This log has a built-in log rotation mechanism that will depart every month. In addition, by default, the log size cannot exceed 10% of the file system size, and the available space of the file system may be lower than 15%. These values can then be/etc/systemd/journald. in conf, the log file size is currently limited to be recorded when the systemd-journald process is started. You can run the following command to view it. This command displays the first two lines of journalctl output:
Journalctl | head-2

You can create the/var/log/journal directory as the root user to change the systemd log to permanent log.
Ensure that the/var/log/journal directory is owned by the root user and the group systemd-journal, and the permission is 2775
Restart the system or send the special signal USR1 to the mongoemd-journald process as the root user.
Killall-USER1 systemd-journal

Because systemd logs are now permanently retained between reboots, you can use journal-B to display only the log messages of the system since the last startup to reduce output.
Journalctl-B-1 # indicates that the output is limited to the previous startup.

Keep accurate time
For analyzing log files between multiple systems, correct synchronization of system time is very important, and the Network Time Protocol (NTP) it is a standard method for computers to provide and obtain correct time information over the Internet. The computer can obtain the correct time information through the public NTP service on the Internet.

Timedatectl # The command briefly displays system settings related to the current event, such as the current system time, time zone, and NTP synchronization settings.
Timedatectl list-timezones # list databases whose systems contain known time zones
Timedatectl set-timezone Time Zone # set the time
Timedatectl set-time 9:00:00 # set the time
Timedatectl set-ntp true | false # enable or disable NTP Synchronization
Chronyc sources-v # verify the NTP server