AWStats Introduction: Download, install, configure samples and use of the log analysis tool for Apache/Windows IIS (including 6.9 Chinese definition patches)

ArticleDirectory

• Download/install
• Configuration File naming rules: AWStats. sitename. conf
• Minimal configuration file modification: logfile sitedomain logformat
• Log Analysis
• Statistical Output
• Automatic Running of log statistics
• Multi-site log statistics
• Statistical indicator description
• Hacking AWStats

You do not have to patiently read all the content: brief installation instructions are as follows:
Install
Http://sourceforge.net/projects/awstats/ download installation package:
GNU/Linux: Tar zxf awstats-version.tgz
AWStats scripts and static files are all in the wwwroot directory by default: deploy all the files in the cgi-bin directory to the cgi-bin/directory: /home/Apache/cgi-bin/AWStats/
MV AWStats-version/wwwroot/cgi-bin/path/to/Apache/cgi-bin/AWStats
Copy the file directories such as icons to the HTML file publishing Directory on the Web, for example,/home/Apache/htdocs /.
More batch update scripts can be stored in the cgi-bin/AWStats/directory under the tools directory.
Upgrade the main search engines and spider definitions in China, and install the geoip application library: c
Http://www.maxmind.com/download/geoip/api/c/ unpack, compile and install
Perl-mcpan-e 'Install "Geo: IP" 'or use a pure Perl Package Perl-mcpan-e' install "Geo: IP: pureperl "'
Download the geoip/geoipcitylite package: unpackage and deploy it to the AWStats directory:

Wget

Http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

Wget

Http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

Configuration
Name the default AWStats. model. conf as common. conf.
Modify some of the configuration options:
Loadplugin = "decodeutfkeys"
Loadplugin = "geoip geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geoip.dat"
Loadplugin = "geoip_city_maxmind geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geolitecity.dat"

Create an AWStats Directory: The data directory is used for statistical data output. The default statistical output is collected on a monthly basis. If the log size is very large, it is best to select daily statistical output and modify AWStats. pl: Set limitflush to 10 times;
$ Limitflush =
50000; # Nb of records in data arrays after how we need to flush data on disk
This can effectively increase the speed of statistics (reduce disk I/O) and avoid excessive statistical files caused by monthly summary of data;

Set the configuration file as follows:
Include "common. conf"
Logfile = "/home/Apache/logs/access_log. % YYYY-24 % MM-24 % DD-24"
Sitedomain = "www.chedong.com"
Hostaliases = "chedong.com"
Defaultfile = "index.html"
Dirdata = "/home/Apache/cgi-bin/AWStats/data /"

Summary: Introduction to AWStats and some configuration improvements. We are glad to see that at the beginning of awstats 6.3, Chinese users basically only need to enable loadplugin = "decodeutfkeys" in the configuration file, so there is basically no statistical problem with the Chinese search engine, currently, multiple major search engine manufacturers in China have been added. Includes patches defined for major search engines and spider in China (unpack and overwrite the original program directory under the Lib \ directory). It also contains the sample configuration file of this site.

The log statistics system plays an important role in site user behavior analysis, especially for keyword access statistics from search engines: it is a very effective data source for user behavior analysis. With the development of the Internet over the years, Web Log statistics tools have become more and more mature and feature-rich. Many of them are openSource codeAWStats is an outstanding one.

AWStats: advanced web statistics

AWStats is a fast-growing Perl-based web log analysis tool on SourceForge. Compared with another excellent Open SourceCodeThe log analysis tool Webalizer and AWStats has the following advantages:

1. User-friendly: You can directly call the corresponding language interface (Simplified Chinese Version) based on your browser)
   Reference output sample: http://www.chedong.com/cgi-bin/awstats/awstats.pl? Config = chedong
2. Perl-based: the system can run on GNU/Linux or Windows (after ActivePerl is installed), and the log analysis directly supports the Apache format (combined) and IIS format (need to be modified ). Although Webalizer has a Windows platform version, it lacks maintenance;
   AWStats allows you to use a system to collect statistics on different Web servers on your website: GNU/Linux/Apache and Windows/IIS servers.
3. High Efficiency: the AWStats output statistics project is much richer than Webalizer, and the speed can still reach about 1/3 of Webalizer. This speed is sufficient for websites with millions of daily visits;
4. Convenient configuration/customization: The system provides flexible but reasonable configuration rules by default. The default configuration to be modified cannot exceed 3 or 4 items to start running, there are also many plug-ins for Modification and extension;
5. AWStats designers are designed for precise "Human visits", so robot access to many search engines is filtered out, therefore, the statistics may be lower than those of other log statistics tools. Access from within the company can also be filtered out through IP address filtering settings.
6. Provides many extended Parameter Statistics functions: using the extraxxxx series configuration to generate Parameter Analysis for specific applications is very useful for product analysis.

For more information about Webalizer and analog, see:
Http://awstats.sourceforge.net/#COMPARISON

AWStats installation memo

The AWStats running mode is as follows:

1. Log Analysis: after running, the log statistics are archived into an AWStats database (plain text;
2. Then there is the output in two forms:
   • One is through CGIProgramReads the output of the statistical result database;
   • One is to run the background script to export the output to a static file;

The following are two log statistics examples for a single site:
One is CGI-based output on GNU/Linux,
One is static page-based export on Windows 2000

Download/install

Http://sourceforge.net/projects/awstats/ download installation package:

GNU/Linux: Tar zxf awstats-version.tgz
By default, AWStats scripts and static files are stored in the wwwroot Directory: deploy AWStats. pl programs to/home/Apache/cgi-bin/AWStats /.
MV AWStats-version/wwwroot/cgi-bin/path/to/Apache/cgi-bin/AWStats
# Copy the icons and other file directories to the HTML file publishing directory of the Web:/home/Apache/htdocs/
More batch update scripts can be stored in the cgi-bin/AWStats/directory under the tools directory,

Windows 2000: Run in Script Mode on the background. Unpack the package and move it to the D: \ AWStats directory.
Copy the icon directory to the publishing directory of IIS: inetpub/icon

Data Source log format and day-based truncation rules

1. For Apache: Set the log format to combined. to truncate logs, you must install the cronolog tool to truncate logs by day:
   Customlog "|/usr/local/sbin/cronolog/path/to/Apache/logs/access_log. % Y % m % d" combined
   For example: logs/access_log.20030326
   Logs are compressed. You can use gzip-D
2. For IIS: by default, there are good daily log truncation rules, but the IIS log format is not suitable for AWStats statistics,
   Therefore, it is best to remove all log fields directly and set them strictly according to the following list:
   • Date
   • Time
   • Customer IP address c-IP
   • User name CS-Username
   • Method CS-Method
   • Uri resource CS-Uri-stem
   • Protocol Status SC-status
   • Number of sent bytes SC-bytes
   • Protocol version CS-version
   • User Agent CS (User-Agent)
   • Refer to CS (Referer)

   Compared with IIS default settings:
   There are:

   • Server IP Address
   • Server Port
   • URI query

   Added:

   • Sent bytes
   • Protocol version
   • Reference

Configuration File naming rules: AWStats. sitename. conf

AWStats. pl automatically calls the configuration file AWStats. sitename. conf of the corresponding Site Based on the site name.
For example, running./AWStats. pl-Config = chedong calls the AWStats. chedong. conf configuration file in the same directory;
If-config is not specified, AWStats. conf or/etc/AWStats. conf in the current directory will be used as the default configuration file.
So it is best to rename the default AWStats. model. conf to AWStats. yoursite. conf; for example: AWStats. chedong. conf,

For statistics on multiple sites, the AWStats configuration file inclusion function is still very useful. We can put General configurations in a document and then use it (Versions later than 5.4 will be supported) the include configuration includes the general configuration in the header of each specific configuration file, and then overwrites the corresponding attributes in the general configuration with other configurations, such:
Include = "common. conf"
Logfile = "/path/to/BBS/access_log"
Sitename = "bbs.chedong.com"

Minimal configuration file modification: logfile sitedomain logformat

For Statistics of Apache logs on GNU/Linux, you only need to modify the two options: logfile sitedomain.

1. GNU/Linux logfile = "/path/to/Apache/logs/access_log. % YYYY-24 % MM-24 % DD-24"
   Windows 2000 logfile = "D: \ iis_logs \ w3sv3 \ ex % YY-24 % MM-24 % DD-24.log"
   This configuration indicates the log file name spelled out by the year, month, and date 24 hours ago;
2. Sitedomain = "www.chedong.com"
   The site name. The default value is null. If it is null, AWStats will refuse to run;
3. You need to modify one more log for IIS statistics:
   Logformat = 2
   The default value is 1: Apache log, and 2 is IIS log.

Other precautions:
Awstats if the SWF file is not filtered, The. SWF file will be converted to pageview. Therefore, if the SWF file on the site is primarily an advertisement, it is best to filter it out:

Log Analysis

./AWStats. pl-Update-Config = sitename-lang = Cn
For example:./AWStats. pl-Update-Config =Chedong
Automatically calls AWStats.Chedong. Conf configuration file

Statistical Output

GNU/Linux http: // localhost/cgi-bin/AWStats. pl? Config = chedong
Windows 2000 http: // localhost/AWStats/awstats.chedong.html

Automatic Running of log statistics

On GNU/Linux: crontab-E: Run at 08:10 every day
# Update AWStats
10 8 *** (CD/path/to/Apache/cgi-bin/AWStats/;./AWStats. pl-Update-Config = chedong)

On Windows 2000: set to run at 08:10 every day
D: \ Perl \ bin \ perl.exe D: \ AWStats \ tools \ awstats_buildstaticpages.pl-Update-Config = chedong-lang = cn-Dir = C: \ Inetpub \ AWStats \-awstatsprog = D: \ AWStats \ wwwroot \ cgi-bin \ AWStats. PL

Multi-site log statistics

AWStats comes with a batch processing tool: Tools/awstats_updateall.pl, which allows you to traverse all configuration files in a directory in batches and run the unified computing. Therefore, the remaining work is mainly due to log synchronization issues.

For multiple sites, many configuration options are repeated. If each configuration file is modified and maintained, it will be very troublesome. AWStats provides the configuration file inclusion function starting from version 5.4, so we can configure a general configuration, such as: Common. conf

The configuration of other sites is as follows: You can overwrite the configuration that is inconsistent with the default one using the following options.
AWStats. bbs. chedong. conf
Include "chedong. Common. conf"
Logfile "/path/to/bbs_log"
Sitename "bbs.chedong.com"

AWStats. www. chedong. conf
Include "chedong. Common. conf"
Logfile "/path/to/www_log"
Sitename "www.chedong.com"
Hostaliases = "chedong.com"

Statistical indicator description

• Visitor: collect statistics based on unique IP addresses of visitors. One IP address represents one visitor;
• Number of visits: A visitor may visit multiple times within one day (for example, once in the morning or once in the afternoon), so within a certain period of time (for example, 1 hour ), number of non-repeated IP addresses, visitor visits;
• Webpage count: the total number of page visits, excluding images, CSS, and JavaScript files. However, if a page uses multiple frames, each frame is regarded as a page request;
• Number of Files: Total number of file requests from the browser client, including images, CSS, JavaScript, etc. The user requests a page, if the page contains images, etc, therefore, the server will send multiple file requests, and the number of files is generally far greater than the number of files;
• Byte: the total data traffic sent to the client;
• Data from Referer: the reference (Referer) field in the log records the address before accessing the corresponding webpage. Therefore, if you click to enter the website through the search results of the search engine, the log contains the user's query address in the corresponding search engine. This address can be parsed to extract the keywords used by the user's query:
  For example:
  15:43:58 123.123.123.123.123-Get/index.html 200 192 HTTP/1.1 Mozilla/4.0 + (compatible; + MSIE + 5.01; + windows + nt + 5.0) http://www.google.com/search? Q = chedong
  AWStats has complete functions in key phrases and keyword statistics of search engines: it can identify more than types of Web Crawlers around the world, it can also identify most mainstream international search engines and local language search engines in many regions.

Hacking AWStats

Install GIS-based plug-ins:

Geoip and GEO: ipfree (AWStats 5.5 +)
Geoip and GEO: ipfree are free of charge. They are more accurate and faster than those obtained through reverse DNS resolution. Geoip APIs are free of charge, and the default database is free of charge. The charge is its Data Update Service. GEO: ipfree not only makes code public, but also makes library data public.

Geoip installation:
Download the C library first: After geoip C is unwrapped
%./Configure; make
# Make install

Download the Perl Library: geoip Perl.
% Perl makefile. pl; make
# Make install

GEO: ipfree installation:
Download GEO: ipfree package
% Perl makefile
% Make
# Make install

Configuration: Enable geoip-related plug-ins in the configuration file:

Loadplugin = "geoip geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geoip.dat"
Loadplugin = "geoip_city_maxmind geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geolitecity.dat"

Maxmind currently provides geoip and geoipcitylite data packets for free: You can regularly download data from the following addresses every month:

Wget

Http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

Wget

Http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

From the recent statistics, it is generally updated on the third day of every month;

References:

• AWStats open source project: http://awstats.sourceforge.net/
• Webalizer open source project: http://www.webalizer.org/
  
• Log Analysis Tool: directory.google.com/top/computers/software/internet/site_management/log_analysis/
• Commercial log statistics/analysis tools: directory.google.com/top/computers/software/internet/site_management/log_analysis/commercial/
  
• Merge logs from multiple sites: www.chedong.com/tech/rotate_merge_log.html
• Log statistics are very important for analyzing the impact of search engines on sites: www.chedong.com/tech/google.html
• AWStats plugin Description: awstats.sourceforge.net/awstats_contrib.html

Author: chelong published on: Last updated on:
Copyright statement: You can reprint it as needed. During reprinting, you must mark the original source and author information of the article and this statement in the form of a hyperlink.
Http://www.chedong.com/tech/awstats.html