Full solution for Web server log statistical analysis (2)

Source: Internet
Author: User
Tags apache log

This completes cronolog configuration and installation. By default, cronolog is installed under/usr/local/sbin.

The command to modify apache Log configuration is as follows:

     
      CustomLog "|/usr/local/sbin/cronolog /www/logs/secfocus/%w/access_log" combined
     


Here, % w indicates that logs are saved in different directories Based on the date and day of the week. In this way, logs of the week are saved. To perform log analysis, you need to copy (or move) the log file every day, if you do not want to save the log for one week, to a fixed location to facilitate processing of the log analysis statistics file, add a scheduled task as follows:

      
       5 0 * * * /bin/mv /www/logs/secfocus/`date -v-1d +\%w`/access_log /www/logs/secfocus/access_log_yesterday
      


In this way, the log statistics and analysis program is used to process the file access_log_yesterday.

For large websites that use Server Load balancer technology, the access logs of multiple servers are merged. in this case, each server cannot use access_log_yesterday when defining or moving log files. Therefore, the server number should be included, such as the server IP address, to distinguish between them. Then run the website image and backup service rsyncd on each server (refer to "use rsync to implement website image and backup", ttp: // www.linuxaid.com.cn/engineer/ideal/article/rsync.htm ), then, download the daily installation configuration file of each server through rsync to the server dedicated for access statistics and analysis for merging.

Merge the log files of multiple servers. For example: log1 log2 log3 and output to log_all:

      
       
Sort-m-t ""-k 4-o log_all log1 log2 log3-m: sort by time using merge optimization algorithm-k 4, -o: stores the sorting result to the specified
       
File.


   V. installation and configuration of the log statistics analysis program webalizer

Webalizer is an efficient and free web server log analysis program. The analysis result is in HTML file format, which allows you to conveniently browse through the web server. Many websites on the Internet use webalizer for web server log analysis. Webalizer has the following features:

1. It is a program written in C, so it has a high operating efficiency. On a machine with a clock speed of 10000 MHz, webalizer can analyze records per second. Therefore, it takes only 15 seconds to analyze a 40 m log file.

2. webalizer supports the standard Common Logfile Format. In addition, it also supports variants of several Combined log formats (Combined Logfile Format, this allows you to collect statistics on the customer's situation and the type of the customer's operating system. Now webalizer supports the wu-ftpd xferlog format and squid log file format.

3. Supports command line configuration and configuration files.

4. You can support multiple languages or perform Localization on your own.

5. Supports multiple platforms, such as UNIX, linux, NT, OS/2, and MacOS.

  

This is the first page of the access statistics analysis report generated by webalizer. It contains the table and bar chart statistics and analysis of the average access volume of each month. Click the minute of each month to obtain detailed statistics for each day of the month.

5.1 Installation

Before installation, make sure that the system has installed the gd library. You can use:

      
       [root@mail root]# rpm -qa|grep gdgd-devel-1.8.4-4gdbm-devel-1.8.0-14gdbm-1.8.0-14sysklogd-1.4.1-8gd-1.8.4-4
      


To confirm that the system has installed two rpm packages, gd-deve and gd.

You can install webalizer either by downloading the source code or directly using the rpm package.

Installing the rpm package is very simple. Find the webalizer package from rpmfind.net and download it later:

      
         rpm –ivh webalizer-2.01_10-1.i386.rpm
      


You can install it.

For the source code method, download the source code package from http://www.mrunix.net/webalizer/, and then install the package:

      
       tar xvzf webalizer-2.01-10-src.tgz
      


There is an lang directory in the generated Directory, which stores various language files, but only the Traditional Chinese version, you can convert it into simplified, or re-translate it yourself. Then enter the generated directory:

      
       cd webalizer-2.01-10./configuremake --with-language=Chinesemake install
      


After compilation, A webalizer executable file is installed in the/usr/local/bin/directory.

5.2 configure and run

To control webalizer running, you can use the configuration file or specify parameters in the command line. The configuration file is simple and flexible, and is suitable for the application environment of automatic web server log statistics and analysis.

The default configuration file of webalizer is/etc/webalizer. conf. When the "-f" option is not used When Webalizer is started, Webalizer searches for the file/etc/webalizer. conf, you can also use "-f" to specify the configuration file (when the server has a virtual host, you need to configure multiple different webalizer configuration files, the webalizer of different virtual hosts uses different configuration files. The configuration options to be modified in the Webalizer. conf configuration file are as follows:

      
       LogFile /www/logs/secfocus/access_log
      


Indicates the path information of the configuration file. webalizer uses the log file as the input for statistical analysis;

      
       OutputDir /www/htdocs/secfocus/usage
      


Used to indicate the directory for saving the generated Statistical Report. We used alias before to enable users to use http://www.secfocus.com/usage/to compile the statistical report.

      
       HostName www.secfocus.com
      


Indicates the host name, which is referenced in the statistical report.

You do not need to modify other options. After the configuration file is modified, You need to periodically generate daily statistical analysis for webalizer.

Run as root: crontab-e enters the scheduled running task editing status and adds the following tasks:

      
       5 0 * * * /usr/local/bin/webalizer –f /etc/secfocus.webalizer.conf15 0 * * * /usr/local/bin/webalizer –f /etc/tomorrowtel.webalizer.conf
      


Here we assume that the system runs two virtual hosts and define the Log Analysis configuration files secfocus. webalizer. conf and tomorrowtel. webalizer. conf respectively. In this way, we define statistical analysis of secfocus logs at a.m. and statistical analysis of tomorrowtel logs at a.m.

Then use the second day.

   6. Protect log statistical analysis reports from unauthorized user access

We certainly do not want our Website access statistics to be browsed by others at will. Therefore, we need to protect the usage directory and only allow access by legal users. The basic authentication mechanism provided by apache can be used here. After configuration, you need to provide the password to connect to this address.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.