How to merge multiple log files

How to merge multiple log files?
The following uses the standard CLF format log (APACHE) as an example:
The log format of Apche is as follows:
% H % L % u % t/"% R/" %> S % B
Example:
111.222.111.222--[03/APR/2002: 10: 30: 17 + 0800] "Get/index.html HTTP/1.1"

200 419

The simplest idea is to read the logs one by one and sort them by the time field in the log.
Cat log1 log2 log3 | sort-K 4-T ""
Note:
-T "": the log field delimiter is a space.
-K 4: sort by 4th fields, that is, [03/APR/2002: 10: 30: 17 + 0800 ].
-O log_all: output to the log_all file.

However, the efficiency is relatively low. If a service already needs Server Load balancer

The number of logs is usually more than 10 million, and the size is several hundred mb. In this way, multiple logs of hundreds of MB need to be arranged at the same time.

Order, machine load can be thought ......
In fact, there is a way to optimize, you know: even if a single log itself is already a "time-based arrangement

And sort provides an optimized merge algorithm for sorting and merging of such files: Use-m

Merge merge option,
Therefore, it is better to merge the three log files log1 log2 log3 in this format and output them to log_all.

:
Sort-m-T ""-K 4-O log_all log1 log2 log3
Note:
-M: Use the merge optimization algorithm

Note: It is best to compress the merged log output and then send it to Webalizer for processing.
Some systems can process 2 GB of files, and some cannot. Some programs can process files larger than 2 GB, and some cannot. Do

Avoid files larger than 2 GB, unless it is confirmed that all programs and operating systems involved in the processing can process such files

. Therefore, if the output file is greater than 2 GB, it is better to zip the log and send it to Webalizer for processing: larger than 2 GB

During file analysis, the possibility of file system errors is relatively high, and gzip can also greatly reduce

I/O operations.

This is how logs are merged in chronological order.