Httpd log and log rotation tool, httpd log Rotation

Httpd log and log rotation tool, httpd log Rotation

Directory:
1.1 Error Log ErrorLog
1.2 access log CustomLog
1.3 log rotation: rotatelogs Tool
1.4 log rotation: cronolog Tool
1.5 comparison between rotatelogs and cronolog

The types of logs to be recorded include Error Log ErrorLog and access log CustomLog. Generally, the default value is used for error logs. You can change the storage path of the error logs at most. Because of the large number of customlogs, you may need to customize them. In fact, ErrorLog is provided by the httpd core module, while CustomLog provides a dedicated module mod_log_config for processing. This module also supports the TransferLog command, which is similar to the CustomLog function and usage, if necessary, please refer to the official manual.

Finally, two log rotation tools are introduced: The rotatelogs tool and cronolog tool that come with apache httpd.

 

1.1 Error Log ErrorLog

Error Log Level: debug, info, notice, warn (default), error, crit, alert, emerg.

Definition Syntax of error logs:

ErrorLog file-path|syslog[:[facility][:tag]]

If file-path is replaced by syslog, the built-in syslog log tool is used. facility is the facility type for recording syslog logs. Generally, this method is not used to record logs.

If file-path is used, two methods are available: Specify the file path directly. If MPs queue is used before, the output log is passed as the standard input to the log processing program after the MPs queue, for example, it is passed to the rotatelogs tool that comes with apache. For example:

ErrorLog "logs/error_log"ErrorLog "|/usr/local/apache/bin/rotatelogs /var/log/error_log 86400" common

Of course, for ErrorLog, the data volume is not too large. Generally, you can directly use file records. For CustomLog, you can use the log Cutting Tool for segmentation, rotation, and other actions.

The error log format is controlled by the ErrorLogFormat command. For example, the following is the default error log format in worker and event modes. For the meanings of each parameter, see ErrorLogFormat.

ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] %7F: %E: [client\ %a] %M% ,\ referer\ %{Referer}i"

1.2 access log CustomLog

Use the CustomLog command to specify the location of the access log record. You can use this command multiple times under the same host to indicate that the same log is recorded in multiple locations. Syntax format:

CustomLog file|pipe format|nickname [env=[!]environment-variable| expr=expression]

You can use LogFormat to specify the log record items, such as whether to record the Client IP address and whether to record the request method. You can also use LogFormat to define log categories (called nickname in httpd terminology), such as common, combined, and combinedio. It supports very flexible record items. For details, see the official manual LogFormat.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedLogFormat "%h %l %u %t \"%r\" %>s %b" common<IfModule logio_module>  # You need to enable mod_logio.c to use %I and %O  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio</IfModule>

Then, use the CustomLog command to use the defined log class. For example:

CustomLog "logs/access_log" combined

Of course, the use of LogFormat to define classification is only for convenience. The CustomLog command can directly define the items to be recorded instead of nickname. For example:

CustomLog "logs/access_log" "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

If the third parameter of CustomLog is provided, that is, env or expr, the logs are recorded only for requests that meet the conditions. For example, this log is recorded independently when a gif file is requested, while other requests are recorded in another log.

SetEnvIf Request_URI \.gif$ gif-imageCustomLog "gif-requests.log" common env=gif-imageCustomLog "nongif-requests.log" common env=!gif-image

Generally, except for special requirements, you only need to use the items provided by LogFormat. Otherwise, Log Customization is very disgusting and complicated.

1.3 log rotation: rotatelogs Tool

The rotatelogs tool is a simple tool provided by apache httpd for log splitting based on time interval and size.

By default, it does not create a log file immediately at startup, but is created only when a request arrives. Likewise, this is also true in the round-robin process. If a time interval is reached and no new request arrives, no new log files will be created for the moment. If the "-f" option is used, log files are created immediately at startup, regardless of whether there are any requests. If the "-c" option is used, a new log file is created no matter whether a new request arrives at each round of replacement.

rotatelogs [ -l ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G)

Option description:

• logfile: Specifies the log record path. Some modifiers can be added, such as. log _ % Y. % m. % d will be in. the suffix of the year, month, and day after the log is appended. If no suffix is specified ". nnnnnnnnnn ", indicating that the creation time point is converted to the total number of seconds from 00:00:00, that is," date + % s ". This suffix is also added for each round replacement. If the relative path is used, it is relative to ServerRoot.
• -n: Use a number as the suffix and represent the round-robin list. For example, "-n 3" indicates that records have been recorded in three files: log.1, log.2, and log.3, no new log files will be created.
• rotationtime: Specifies the time interval of rotation. Initialize the start value of the corresponding time format. For example, if the value is set to 3600, the rotation is performed at the beginning of each hour. Although the current time point may only take 5 minutes to enter the next hour, the rotation is also performed after 5 minutes.
• filesize: Replace logs in a large or small way.
• -l: Calculate the time interval using the local time. The UTC time is used by default.
• -f: Forcibly open the log file immediately. In some cases, when httpd is started, no requests may arrive within a short period of time. Because no logs need to be recorded, no log files will be created for the moment. Use this option to create a file immediately.
• -t: Truncate logs instead of replacing logs. In this case, no file suffix is added.
• -v: Detailed records of the rotation or truncation information.
• -e: Outputs logs to the standard error output. This option is useful when logs need to be processed by other tools.
• -c: A New file is created at each interval, although no logs are generated. By default, if no logs arrive at the polling interval, no logs will be created temporarily, but they will not be created until the first request arrives.

For more time class modifiers, see the following.

Note: if there is a duplicate file name during rotation, for example, the rotation is performed at 5 MB, but the file name format is ". log _ % Y. % m. % d ", if there are multiple replicas in the same day, because the file name is the same, it will overwrite the old file for record.

In addition, rotatelogs can only use a time modifier for the log file name itself, and cannot be set to a directory. Otherwise, an error is reported because the log cannot be opened when httpd is started.

Example: The following uses a log Pipeline "|", indicating that the recorded logs are passed to the subsequent Program (rotatelogs here) for processing.

Replace $ ServerRoot/logs/mylog. nnnnnnnnnn every day. nnnnnnnn is the number of seconds converted from the current time. Log files are not created immediately at startup, and they are not created immediately at rotation.

CustomLog "|/usr/local/apache/bin/rotatelogs logs/mylog 86400" common

Rotate by file size.

CustomLog "|bin/rotatelogs /var/log/logfile 5M" common

The Rotation Error Log is replaced when it reaches 5 MB.

ErrorLog "|bin/rotatelogs /var/log/errorlog.%Y-%m-%d-%H_%M_%S 5M"

Truncate logs rather than replace logs.

CustomLog "|bin/rotatelogs -t /var/log/logfile 86400" common

The following are the modifiers of the date class. The following cronolog tool also uses the same modifier.

% CHARACTER n line feed t horizontal tab time class: H hour (00 .. 23) I hour (01 .. 12) p the AM or PM in the locale identifies M minutes (00 .. 59) S seconds (00 .. 61, which allows for leap seconds) X time identifier (e.g.: "15:12:47") Z time zone. If the time zone cannot be determined, there is no significance for the date class: a short name of the working day under the locale (e.g.: Sun .. sat) A full name of the working day under the locale (e.g.: Sunday .. satur-ay) B. abbreviation of the month under the locale (e.g.: Jan .. dec) B. The full name of the month under the locale (e.g.: January .. december) c the date and time (e.g.: "Sun Dec 15 14:12:47 GMT 1996") d days in the current month (01 .. 31) j. Days in the current year (001 .. 366) m months (01 .. 12) the number of weeks in the year U starts from Sunday as a week. The first week is the week with Sunday (00 .. 53) W. the number of weeks in the current year starts from Monday as a week. The first week is the week containing Sunday (00 .. 53) Number of working days (0 .. 6, 0 indicates Sunday) x indicates the date under the locale (e.g. "13/04/97") y two-digit year (00 .. 99) Y four-digit year (1970 .. 2038)

1.4 log rotation: cronolog Tool

You can download it from the epel source or install it on github: https://github.com/fordmason/cronolog.

[root@xuexi ~]# cronolog -husage: cronolog [OPTIONS] logfile-spec   -H NAME,   --hardlink=NAME maintain a hard link from NAME to current log   -S NAME,   --symlink=NAME  maintain a symbolic link from NAME to current log   -P NAME,   --prev-symlink=NAME  maintain a symbolic link from NAME to previous log   -l NAME,   --link=NAME     same as -S/--symlink   -h,        --help          print this help, then exit   -p PERIOD, --period=PERIOD set the rotation period explicitly   -d DELAY,  --delay=DELAY   set the rotation period delay   -o,        --once-only     create single output log from template (not rotated)   -x FILE,   --debug=FILE    write debug messages to FILE                              ( or to standard error if FILE is "-")   -a,        --american         American date formats   -e,        --european         European date formats (default)   -s,    --start-time=TIME   starting time   -z TZ, --time-zone=TZ      use TZ for timezone   -V,      --version         print version number, then exit

This tool is easy to use, but you must understand its rotation principle. The following settings are used as an example:

CustomLog "|/usr/local/sbin/cronolog logs/%Y/%m/%d/access.log" combined

Because the relative path is used, it is relative to ServerRoot. Assume that ServerRoot is/usr/local/apache, the logs here will be created in/usr/local/apache/logs/% Y/% m/% d/access. log, where % Y, % m, and % d indicate year, month, and day, respectively. A file name or directory composed of these modifiers is called a template.

Cronolog rotation principle is: Compare the current time point of the current log file template with the current time. If a part of the template is different from the current time point, the rotation is required, during the rotation, the missing directory is automatically created and the time point for the next rotation is calculated. During rotation, first close the current log file, then create a new log file based on the current time point, and open the new log file. However, note that the cronolog tool does not immediately create a log file if no request arrives during creation or rotation. This is the same as the rotatelogs tool by default.

It looks simple. It is really simple. It only needs to be compared with the current time point based on the specified time modifier. For example, in the above configuration, if the current time is, it will be created in the following order. Note that it will only be created when the request arrives and will not be created when no request is available.

• Period 1: created when the first request arrives
  • Check whether the logs/2017 directory exists. If it does not exist, create it.
  • Check whether the logs/2017/10 directory exists. If it does not exist, create it.
  • Check whether the logs/logs /10/01 directory exists. If it does not exist, create it.
  • Create the logs/2017/10/01/access. log File and write logs.
• Period 2: created when the first request arrives at AM
  • The logs/2017 directory already exists and is not created.
  • The logs/2017/10 directory already exists and is not created.
  • The logs/logs /10/02 directory does not exist and is created.
  • Create the logs/2017/10/02/access. log File and write logs.

The rule will be followed.

That is to say, cronolog uses the time modifier of the smallest unit as the rotation interval by default. For example, if the minimum unit is % d, it is replaced by a daily round. If the minimum unit is % W, it is replaced by a weekly round. If the minimum unit is % S, it is replaced by a second. However, it must be noted that only when the request arrives will a new log file be created based on the current time point. For example, if a log file is created in 10th seconds for the second round, the log file will be closed in 11th seconds, but no new log file will be created immediately. If a new request arrives in 15th seconds, create a 15.logstore instead of 11.log.

In addition, cronolog can use the "-p N UNITs" option to explicitly specify the rotation interval. The effective UNITs of UNITs are seconds, minutes, hours, days, weeks, and months. For example, "-p 5 minutes" indicates a replacement every 5 minutes. However, note that the value of N must be the public factor (excluding the maximum public factor) of a higher level of UNITs. For example, if the length of an hour is 60 minutes, you can specify "1, 2, 3, 4, 5, 6, 10, 15, 20, 30 minutes ", but cannot be specified as" 7, 9, 11, 12, 13, 14 "and other minutes.

If you do not want to replace, there are two ways: Do not use the time class modifier; use the "-o" option. At this time, the same file will always be written.

Cronolog also supports the dynamic symbolic link function, such as the following configuration, so that each access to/usr/local/apache/logs/access_log can access the latest log files.

CustomLog "|/path/to/cronolog --symlink=/usr/local/apache/logs/access_log /usr/local/apache/logs/%Y/%m/access_log" combined

Finally, cronolog does not support log truncation.

1.5 comparison between rotatelogs and cronolog

Both tools have their own advantages and disadvantages.

In fact, these are all minor issues.

Go back to the Linux series article outline: workshop!