Customize Apache Log rollback tools in Embedded Systems

Customize the log rollback tool of Apache in the embedded system-Linux Enterprise Application-Linux server application information. The following is a detailed description. The most widely used web servers in the world. With the widespread use of Apache, the more accesses, the more logs, and how to effectively manage logs is a very important issue, therefore, it is necessary to select a good log Manager program.

Introduction

Currently, many embedded devices provide a Web access mode to facilitate the management and configuration of embedded devices. Apache server is currently the most widely used Web server. To effectively manage Web servers and master and feedback Server Activity, performance, and problems, we usually open the log files (access logs and error logs) of Web servers) to facilitate subsequent queries and statistics. Although the current embedded devices have greatly improved their performance and storage capacity, compared with hard disks, the flash storage capacity of embedded devices is still small. Even if a server is not busy, after a long period of operation, the log file will become larger and larger. If no processing is performed, the log file will affect the running efficiency and speed of the Web server, and may exhaust the small storage space of embedded devices, resulting in system slowdown or even failure to run normally, therefore, it is necessary to regularly roll back log files. How to customize log processing programs to manage logs is the focus of this article.




Apache Log Mode

The Apache server has two ways to enable logs: Normal Mode and pipeline mode.

1) Common log usage mode

By default, the Apache server automatically opens two log files: access log files and error log files. In the configuration file httpd. the command "CustomLog/usr/local/apache/logs/access_log common" and the command "ErrorLog/usr/local/apache/logs /error_log common "to indicate. The error log file is more important. The Apache server stores diagnostic information and errors in request processing in this file for future error diagnosis and resolution. By default, log files generated in this way are not separated and become larger over time. Such a large file is neither easy to manage nor easy to analyze and collect statistics in the future. Therefore, it is necessary to regularly roll back log files.

2) MPs queue Mode

The latest Apache server version provides a new log processing method, that is, the server does not directly write logs into a file, instead, the access record and error information are transmitted to another process, usually known as log rollback, which is divided (rolled back) according to user settings ). The implementation principle is as follows:

Figure 1. Schematic diagram of MPs queue mechanism
Schematic diagram of MPs queue mechanism

This mode does not require programming the master server, and allows log rollback without restarting the server. This function significantly enhances the flexibility of logs.

The pipeline mode is relatively simple. You only need to use the pipeline operator "|" in the 'customlog' or 'errorlog' command of the configuration file, then, a rollback tool file path is provided, so that the program can obtain the log records from the standard input device. Apache provides a reliable pipeline log technology. When it is started, the pipeline log process is started at the same time. If the rollback process crashes during running, the process is restarted.




Popular log rollback tools

The following describes the three most popular Log File rollback tools, briefly describes their usage, and analyzes their respective advantages, disadvantages, and applicable scenarios.

1) logrotate Tool

The logrotate tool is a log rollback program that comes with the Linux system. It is started by the cron program of the System Planning Service tool and can be used to roll back various system logs. For web server log management, you only need to add the corresponding commands in the logrotate. conf configuration file.

This is a way to roll back log files without using other third-party tools. This tool can roll back logs by specifying the rollback time or log file size, and you can also specify the number of backup files to be retained. It is a good choice for embedded systems with limited storage space. However, the logrotate tool has a serious disadvantage, that is, by sending the HUP restart command to web server processes to truncate the current log file and perform round robin. Log rollback in this way will seriously affect the continuity of web Services, so it is not practical for real-time monitoring systems.

2) rotatelogs Tool

The rotatelogs tool is a simple log rollback program that comes with Apache. You can start it by adding the following command to the Apache configuration file: errorlog "|/usr/local/apache2/bin/rotatelogs/usr/local/apache2/logs/apache_error_log 86400 ".

This is a way to roll back log files without restarting the server through pipelines, greatly enhancing the log processing capability. This tool allows you to control logs on time or by file size. When a log file is rolled back by time, the specified time refers to the rollback interval and is relative to the server startup time. That is, if the web server is restarted midway through, a new log file will be generated, and the interval between the newly generated file and the last generated file is not a set rollback interval, this may affect subsequent Log File analysis programs. During file rollback, the minimum size of the specified file is in megabytes. This is not suitable for embedded systems with limited storage. In addition, the rotatelogs tool does not delete old log files or specify the number of old rollback log files to be retained. Therefore, if the generated old log files are not transferred or deleted in time, the storage space will be occupied more and more, which is not suitable for embedded systems.

3) cronolog Tool

Cronolog is a simple, compact, and efficient log file rollback tool. It is started in the same way as rotatelog, that is, ErrorLog "|/usr/sbin/cronolog/web/logs/% Y/% m/% d/errors. log ". However, the name of the generated log file is composed of the file name template in the instruction and the current date and time.

The cronolog tool can only roll back and forth log files by time, but supports roll back logs by year, month, or day. You can also set the format of the directory structure to clearly store log files. For example, if % Y/% m/% d/error_log is specified, error_log will be stored in the directory of year/month/day. However, this tool does not delete old log files, which increases storage usage. Therefore, it is not suitable for embedded systems.




How to customize log rollback tools

Because the above tools are not suitable for embedded systems, we will explain in detail and implement a log tool that meets the principles of the log rollback tool and meets the requirements of embedded systems. This tool provides interfaces similar to other log tools. You can set the log file size and retain the number of old logs.

System preparation:

First, we can see from the principles of the Apache pipeline mode that we need to use the pipeline operator "|" to start the pipeline mechanism of the Apache server. Therefore, to ensure proper running, make sure that the current system contains shell script actuators. Generally, the smallest linux system is used in embedded systems. The busybox command tool is used to provide the most common system commands. To be similar to the linux system on a PC, we recommend that you create a sh link file in the/bin folder.

Second, many file systems in embedded systems use read-only file systems. Therefore, to ensure successful creation of log files, make sure that the file system you want to save logs is writable.

Principles and implementation:

Because the Apache server will output the log information to the screen and convert it into a standard input through a pipeline. Therefore, log rollback first needs to read the output log information from the screen, where 0 indicates the standard input file descriptor.

N_log = read (0, log_buf, sizeof (log_buf ));


Second, before writing the read log information to a file, you need to determine whether there is a log file available for writing. If the log size does not reach the specified file size, the log is written to the file. Otherwise, create a log file.

If (log_fd> = 0)
{
If (filesize (log_fd)> = f_size)
{
Fclose (log_fd );
Log_fd =-1;
}
}
If (log_fd =-1)
{
Log_fd = create_new_log ();
}


Finally, when everything is ready, we can safely write this information into the log file.

Write (log_fd, log_buf, n_log );
N_log = read (0, log_buf, sizeof (log_buf ));


In this way, the log information is registered and the next log operation is started again. The implementation process is as follows:

Log rollback tool Flowchart

Figure 2. Log rollback tool Flowchart
Log rollback tool Flowchart

The preceding describes a common log rollback tool suitable for embedded systems. Because of the large differences or different requirements of embedded systems, users can customize targeted log tools based on their actual applications.