Multi-server log merge statistics-cronolog round-robin of Apache logs

Abstract: You have no patience to read all the following content, because the conclusion is nothing more than the following two points:
1. Use cronolog to clean and securely track Apache logs
2. Merge and sort multiple logs with sort-m
Or use clfmerge to merge logs.

Based on personal experience:
1. First introduce how to merge Apache logs;
2. Describe the necessity and solution of log round-robin based on the problem, and describe how to use cronolog to round-robin Apache logs;
There are a lot of related tool usage skills and some failed attempts in the design log merging process ......
I believe that there are more than one way to solve the above problems. The following solutions are certainly not the easiest or the lowest cost. I hope to have more exchanges with you.

Necessity of multi-server log merge statistics

More and more large Web services use DNS round robin to achieve load balancing: using multiple servers with the same role as the front-end Web Services greatly facilitates service distribution planning and scalability, but multiple services
The distribution of the server makes log analysis and statistics a little effort. If you use Webalizer and other log analysis tools to make separate log statistics for each machine:
1 will bring a lot of trouble to the Data Summary. For example, the total access volume of the statistics needs to add the number of the specified month on server1 server2.
2. The unique visitor number unique visits and unique website number unique will be greatly affected.
Sites and other indicators, because these indicators are not the algebra of several machines.

The benefits of unified log statistics are obvious. But how can we combine the statistics of all machines into a statistical result?
First, you may think: Can multiple servers record logs to the same remote file? We do not consider using a remote file system to record logs, because the trouble is much more convenient than what you get ......
Therefore, the logs of multiple servers to be counted are recorded separately and regularly synchronized to the backend through a certain method => merge => and then analyzed using the log analysis tool.

First, we need to explain why logs should be merged: Because Webalizer does not have the function to merge multiple logs on the same day
Run successively
Webalizer log1
Webalizer log2
Webalizer log3
The final result is: Only log3 results.

Can I simply overlay log1 <log2 <log3?

Because a log analysis tool does not read all the logs at a time for analysis, and the stream reads the logs and saves the staged statistical results at a certain interval. Therefore, the time span is too large (for example, two logs
If the interval exceeds 5 minutes, the algorithms of some log statistics tools will "forget" the previous results ". Therefore,
The statistical result of log1 <log2 <log3 direct file connection is: only the statistical result of log3.

Multiple services

Log merging: sorts records in multiple logs by time and then merges them into one file.

The typical time fields of multiple log files are as follows:
Log1 log2 log3
00:15:00 00:14:00

00:16:00 00:15:00
00:17:00 00:18:00
00:18:00 00:19:00
14:18:00 11:19:00
15:18:00 17:19:00
23:18:00 23:19:00

Logs must be merged by time. The merged logs should be:
00:15:00 from log1

00:15:00 from log2
00:16:00 from log1
00:17:00 from log3
00:18:00 from log2
00:19:00 from log1
....

How to merge multiple log files?
The following uses the standard CLF format log (APACHE) as an example:

The log format of Apche is as follows:
% H % L % u % t/"% R/" %> S % B
Example:
111.222.111.222--[03/APR/2002: 10: 30: 17 + 0800] "Get/index.html
HTTP/1.1 "200 419

The simplest idea is to read the logs one by one and sort them by the time field in the log.
Cat log1 log2 log3 | sort-K 4-T ""
Note:
-T "": the log field delimiter is a space.

-K 4: sort by 4th fields, that is, [03/APR/2002: 10: 30: 17 + 0800 ].
-O log_all: output to the log_all file.

However, the efficiency is relatively low. If a service already needs Server Load balancer, the number of stand-alone logs of the service is usually more than 10 million, with the size of several hundred mb. In this way, the number of logs of several hundreds of MB is required at the same time.
Sort the server load ......
In fact, there is a way to optimize it. You must know that even if a single log is already a file named "sorted by time, sort provides an optimized merge algorithm for sorting and merging such files:
Use
-M merge option,
Therefore, it is better to merge the three log files log1 log2 log3 in this format and output them to log_all:
Sort-m-T ""-K 4-O log_all log1 log2 log3
Note:
-M: Use the merge optimization algorithm

Note: It is best to compress the merged log output and then send it to Webalizer for processing.
Some systems can process 2 GB of files, and some cannot. Some programs can process files larger than 2 GB, and some cannot. Avoid files larger than 2 GB as much as possible, unless it is confirmed that all programs and operating systems involved in the processing can process
Such a file. Therefore, if the output file is greater than 2 GB, it is better to zip the log and send it to Webalizer for processing: The file system error may be more likely during file analysis process larger than 2 GB,
In addition, Gzip can greatly reduce I/O operations during analysis.

This is how logs are merged in chronological order.

Log rotation mechanism

Let's take a look at the data source: Webalizer is actually a tool for monthly statistics and supports incremental statistics: for large services, I can combine Apache logs by day and send them
Webalizer statistics. How does one truncate Web logs by day (for example, 00:00:00 every night?

If you use crontab every day: Back up logs to access_log_yesterday at every day
MV/path/to/Apache/log/access_log
/Path/to/Apache/log/access_log_yesterday
You also need to run Apache restart immediately.
Otherwise, Apache will not know where to record the log because of the loss of the log file handle. In this way, the apache service will be affected when the archive is restarted every night.
A simple method that does not affect services is to copy and then clear
CP/path/to/Apache/log/access_log
/Path/to/Apache/log/access_log_yesterday
Echo>/path/to/Apache/log/access_log

Serious analysts will find a problem:
However, it is impossible for CP to strictly guarantee zero-point truncation. It takes 6 seconds to add the data to the copy process. The copy process is up to 00:00:06 In the truncated access_log_yesterday log.
Between logs. It is no problem to count the hundreds of lines of logs generated each day in a single log. However, there will be a Merge Sorting Problem for multiple logs within one day of the new month:

[31/MAR/2002: 59: 59: 59 + 0800]
[31/MAR/2002: 23: 59: 59 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]

You need to know [01/APR/2002: 00: 00: 00
This field cannot be sorted across days. Because dd/mm/yyyy, month or English name are used in the date, if you sort by letter, it is likely that the result is: Sorting causes logs
Error
[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]

[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]
[01/APR/2002: 00: 00: 00 + 0800]
[31/MAR/2002: 59: 59: 59 + 0800]
[31/MAR/2002: 59: 59: 59 + 0800]
[31/MAR/2002: 23: 59: 59 + 0800]
[31/MAR/2002: 59: 59: 59 + 0800]
[31/MAR/2002: 23: 59: 59 + 0800]

These abnormal data during the cross-day process is like eating a bug for analysis tools such as Webalizer. The running result is: it may lose all the data in the previous month! Because
This data may cause many risks in the process of processing data from the last day of the previous month.

There are several ways to solve the problem:
Post-processing:
. Therefore, you can use the grep command to remove logs from the previous month on the first day of each month. For example:
Grep-V "01/APR" access_log_04_01> access_log_new

Modify the sort log: Remove all cross-day data. The post-processing of logs may be a way, although the sort command has special options for sorting dates
-M (Note: uppercase m) allows the specified fields to be sorted by the English month rather than the alphabetic order. However, it is very troublesome to use the sort command to split the month field in Apache logs. (I tried to use
"/" Is used as the delimiter and "month" is used"
(Year: Time ). Although some Perl scripts can be implemented, I finally gave up. This does not comply with the system administrator design principles: versatility.
And you need to keep asking yourself: Is there a simpler way?

Another is to change the log format to timestamp (for example, the squid log does not have this problem, and its log itself uses timestamp for timestamp), but I cannot guarantee
All log tools can recognize that you have used a special format for the date field.

2. Optimize the Data source:
The best way is to optimize the data source. Ensure that the data source is rotated by day, and the data in the logs of the same day is within the same day. In this way, no matter what tool you use (commercial, free) to analyze logs
This is because the complex log preprocessing mechanism is affected.

The first thing you may think of is to control the log truncation time. For example, to strictly capture logs from, but there is no difference between intercepting logs one minute before midnight or one minute later, you still cannot control the cross-
Problems recorded in 2 days, and you cannot predict the time used in the log archiving process.
Therefore, you must consider the use of log tracking tools. These log tracking tools must meet the following requirements:
1. Do not interrupt Web Services: Do not stop Apache => Mobile Log => restart Apache
2. Ensure that logs on the same day can be rotated by day: One log every day 00:00:00-23:59:59

3. It is not affected by Apache restart: If Apache is restarted every time, a new log is generated, which does not meet the requirements.
4. Easy installation and configuration

First, we consider the built-in round robin tool rotatelogs under the Apache/bin directory.
This tool is basically used to control logs on time or by size, and cannot control when to truncate and how to archive logs by day.
Then consider the logrotate Background Service: logrotate is a background service dedicated to round-robin of various system logs (Syslogd, mail), such as system
Log, but its configuration is complicated and abandoned. In fact, it also sends a-hup restart command to the corresponding service process to archive log truncation.

In the Apache FAQ, we recommend a mature tool named cronolog that has been developed for nearly two years. The installation is simple: Configure => make =>
Make install

One of his configuration examples will show you how suitable it is for logging round-robin by day: Just make a small modification to httpd. conf:

Transferlog "|/usr/sbin/cronolog/web/logs/% Y/% m/% d/access. log"
Errorlog "|/usr/sbin/cronolog/web/logs/% Y/% m/% d/errors. log"

Then: the log will be written
/Web/logs/2002/12/31/access. Log
/Web/logs/2002/12/31/errors. Log
After midnight: logs will be written
/Web/logs/2003/01/01/access. Log
/Webpage/logs/2003/01/01/errors. Log

And 2003 2003/01 and 2003/01/01 will be automatically created if they do not exist.

Therefore, as long as you do not adjust the system time at 00:00:00, logs should be fully stored on a daily basis (23:59:59-), followed by log analysis:
The field [31/MAR/2002: 15: 44: 59 has nothing to do with the date. It is only related to the time.

Test: Taking into account the system hard disk capacity, it is decided to cycle logs by week
Add the following to Apache configuration:
# % W weekday
Transferlog "|/usr/sbin/cronolog/path/to/Apache/logs/% W/access_log"

After Apache is restarted
/Path/to/Apche/logs/access_log is created under the system log directory.
3/directory (tested on week 3). After a while, I suddenly found that the growth rates of the two logs were different!

We can only find two logs after tail:
I set customlog TO USE THE COMBINED format, that is, containing (Extended information), while transferlog uses the default log format.
It is known that transferlog uses a format closest to itself in the configuration file as the log format. What I wrote in httpd. conf is:
Logformat... combined
Logformat... common
...
Customlog... combined
Transferlog...

Therefore, trasferlog uses the default format, which is required for transfer logs to use the specified format:

Logformat "% H % L % u % t/" % R/"%> S % B/" % {Referer} I /"
/"% {User-Agent} I /""
Transferlog "|/usr/local/sbin/cronolog
/Path/to/Apache/logs/% W/access_log"

Restart, OK, and the log format is the same.
In fact, this setting Result Records two logs access_log and % W/access_log in the logs directory respectively. Can I only record the logs under % W?
Query the Apache manual. A simpler method is to directly output the customlog to the cronolog archive log and specify the format.
Customlog "|/usr/local/sbin/cronolog
/Path/to/Apache/logs/% W/access_log "combined

Finally, there is a log synchronization problem.

Task: Find the log of the previous day in the early morning, and save another file to be sent to the server.
For example, if you want to keep the logs of the previous week: copy the logs of the previous day to the specified directory every day and wait for the log server to capture the logs:
/Bin/CP-F/path/to/Apache/logs/'date-v-1d + % W'/access_log
/Path/For/backup/logs/access_log_yesterday

Use the following command on FreeBSD
Data-v-1d + % W
Note:
-V-1d: the previous day, and on GNU/Linux this option should be date-d yesterday
+ % W: weekday. All tools use the same weekday definition as the standard time function library. 0-6 => Sunday-Saturday

Note:
Add a "/" escape before "%" when writing to crontab: logs are archived at 00:05 every day,
Another problem is to use Rm-f {} Instead of RM-f {}/In cront {}/;
5 0 ****/bin/CP/path/to/logs/'date-v-1d +/% W'/access_log
/Path/to/for_sync/logs/access_yesterday
37 10 ***/usr/bin/find/home/Apache/logs/-name access_log-mtime + 1
-Exec/bin/Rm-f {};

The first time cronolog log is counted as 3 weeks. After one week, the log returns 3/access_log.
But will this log be appended to 3/access_log or a new file?> Access_log or> access_log?

My test result is that the log will be appended:
[01/APR/2002: 23: 59: 59 + 0800]
[01/APR/2002: 23: 59: 59 + 0800]
[08/APR/2002: 00: 00: 00 + 0800]
[08/APR/2002: 00: 00: 00 + 0800]

Certainly, I do not want to repeat the statistics of last week's data every time (although it does not affect the results), and is there more and more logs under % W?
Solution 1: change the daily CP to mv
Solution 2: After the data is copied every day, the access_log logs generated six days ago are deleted.

Find/path/to/Apache/logs-name access_log-mtime + 6-exec Rm-f {}/;
It is necessary to retain logs for several days: What if the Log Analysis Server breaks down for one day?

The following is a daily statistics script file that installs Apache under/home/Apache:
#! /Bin/sh

# Backup old log
/Bin/CP-f/home/Apache/logs/'date-d Yesterday + % W'/access_log
/Home/Apache/logs/access_log_yesterday

# Remove Old log
/Usr/bin/find/home/Apache/logs-name access_log-mtime + 6-exec Rm-F
{}/;

# Analysis With Webalizer
/Usr/local/sbin/Webalizer

Summary:
1. Use cronolog to clean and track logs safely
2. Merge multiple logs by sort-M sorting

References:

Log analysis and statistics tools:
Http://directory.google.com/Top/Computers/Software/Internet/Site_Management/Log_Analysis/

Apche log settings:
Http://httpd.apache.org/docs/mod/mod_log_config.html

Apache Log cycle:
Http://httpd.apache.org/docs/misc/FAQ.html#rotate

Cronolog
Http://www.cronolog.org

Webalizer
Http://www.mrunix.net/webalizer/

Webalzer for Windows
Http://www.medasys-lille.com/webalizer/

Introduction to AWStats
Http://www.chedong.com/tech/awstats.html

Author: chelong District
Posted on: Last updated on:
Copyright Notice
: You can reprint it as needed. During reprinting, you must mark the original source of the article in the form of a hyperlink.
And author information and this statement
.
Http://www.chedong.com/tech/rotate_merge_log.html