Multiple methods for splitting WebServer logs

Multiple methods for splitting WebServer logs-Linux Enterprise Application-Linux server application information. For more information, see the following. The complete documentation for this article is: http://dev.weamax.com/docs/books/log_process/log_process.html

Author: Kenny Zhao

Why split logs?
As the website access grows, the log files generated by WebServer become larger and larger. If logs are not separated, you can only delete large logs (such as Apache logs) at one time, in this way, a lot of valuable information about the website is lost, because these logs can be used for access analysis, network security monitoring, and network operation status monitoring, therefore, managing these massive logs is of great significance to the website.

This article will summarize some practical log segmentation methods, hoping to facilitate the Linux/Unix administrator to effectively manage log files. If too many files are large, you can delete some historical files.



Several log segmentation methods
2.1.1. Use a third-party program cronolog for log Segmentation
2.1.2. Use a self-written script to separate logs
2.1.3. Use a third-party program newsyslog to separate logs

2.1.1. Use a third-party program cronolog for log Segmentation
This method is the most convenient and fast, but it only supports Apache.


Example 2.1. Use cronolog to separate logs

1. Download cronolog

Http://cronolog.org/download/cronolog-1.6.2.tar.gz

Tar xzvf cronolog-1.6.2.tar.gz
Cd cronolog-1.6.2
2. Compile


./Configure
Make
Make install
3. Modify the Apache configuration file httpd. conf and add the following code:


CustomLog "|/usr/local/sbin/cronolog/var/log/httpd/www/access % Y % m % d. log" combined
Note:
If Apache has multiple virtual hosts, it is best to place such code in each virtual host and change the log file name to a different name.



Prompt
/Var/log/httpd/www/can be changed to any log output path


4. Save the configuration and restart the Apache service.


/Etc/rc. d/init. d/httpd stop
/Etc/rc. d/init. d/httpd start
The last generated file name is in the following format:

/Var/log/httpd/www/access20050918.log


2.1.2. Use a self-written script to separate logs
If cronolog splitting is not allowed due to the condition, a more concise method is provided, which is suitable for the following systems:

Linux
Unix
* BSD

Example 2.2. Generate a file every day and compress and store it as scheduled tasks

Note:
Assume:/var/log/httpd/is the log storage path, and access_log is the previous log file name.

1. Create a new file:/var/log/httpd/log-task.cron

The file content is as follows:


#! /Bin/bash

YESTERDAY = 'date-d yesterday + % Y % m % d'
/Usr/bin/gzip-c/var/log/httpd/access_log>/var/log/httpd/access?#yesterday=.log.gz
>/Var/log/httpd/access_log
Save and exit


2. Modify the execution permission of the file and test the script.

CODE:
[Copy to clipboard]
Chmod u + x/var/log/httpd/log-task.cron
Run the script once immediately to test whether the script works properly.


Note:
We recommend that you back up the log file (/var/log/httpd/access_log) before testing)


Test:

/Var/log/httpd/log-task.cron.
Check whether the file named by date has been generated in the/var/log/httpd directory.

3. Add the script to the automatic running task.

Crontab-e

Then copy the following code to the end of the cron STARTUP script:

1 0 ***/var/log/httpd/log-task.cron
Save and exit.

Prompt
(The system will automatically perform a log rollback task at 00:01 every day. A compressed log file is generated every day under the/var/log/httpd directory. The file name is access20051216.log.gz)


2.1.3. Use a third-party program newsyslog to separate logs
This method applies to SQUID and other webservers that cannot use cronolog. The disadvantage is that the installation is complicated.

Example 2.3. Use newsyslog to separate logs

Note:
Assume:/usr/local/squid/var/logs/is the log storage path, and access. log is the previous log file name.

1. Download newsyslog

Http://archives.eyrie.org/software/system/newsyslog-1.8.tar.gz


Tar xzvf newsyslog-1.8.tar.gz
Cd newsyslog-1.8
2. Compile

./Configure
Make
Make install
3. Create a new file/usr/local/etc/newsyslog. conf

Set squid_logpath =/usr/local/squid/var/logs
Set squid_log =/usr/local/squid/var/logs/access. log
Set date_squid_log =/usr/local/squid/var/logs/access % Y % M % D. log

SQUID {
Restart: run/usr/local/squid/sbin/squid-k rotate
Log: SQUID squid_log squid 644
Archive: SQUID date_squid_log 0

}
Save and exit


Prompt
/Usr/local/squid/var/logs can be changed to any log output path


4. Add the script to the automatic running task.

Crontab-e

Then copy the following code to the end of the cron STARTUP script:

CODE:
[Copy to clipboard]
58 23 ***/usr/local/sbin/newsyslog
The last generated file name is in the following format:

/Usr/local/squid/var/logs/access20050918.log