AWStats Introduction: A log analysis tool for Apache/IIS

ArticleDirectory

• Download/install
• Configuration File naming rules: AWStats. sitename. conf
• Minimal configuration file modification: logfile sitedomain logformat
• Log Analysis
• Statistical Output
• Automatic Running of log statistics
• Multi-site log statistics
• Statistical indicator description
• Hacking AWStats

you do not have to patiently read all the content: brief installation instructions
Installation
after the http://sourceforge.net/projects/awstats/ download the installation package:
GNU/Linux: Tar zxf awstats-version.tgz
AWStats scripts and static files are both under the wwwroot directory by default: deploy all the files in the cgi-bin directory to the cgi-bin/directory: /home/Apache/cgi-bin/AWStats/
MV AWStats-version/cgi-bin/path/to/Apache/cgi-bin/AWStats
put icons, etc. copy the file directory to the HTML file publishing Directory on the Web, for example, release more batch update scripts under/home/Apache/htdocs/in the tools directory, can be put together in cgi-bin/AWStats/directory
Upgrade domestic main search engine and spider definition, install geoip application library: C
http://www.maxmind.com/download/geoip/api/c/ unpack, compile and install
Perl-mcpan-e 'Install "Geo: IP" 'or use a pure Perl Package Perl-mcpan-e' install "Geo: IP :: pureperl "'
download geoip/geoipcitylite package: unpack and deploy to AWStats directory
http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz

Configuration
Name the default AWStats. model. conf as common. conf.
Modify some of the configuration options:
Loadplugin = "decodeutfkeys"
Loadplugin = "geoip geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geoip.dat"
Loadplugin = "geoip_city_maxmind geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geolitecity.dat"

Create an AWStats Directory: data directory for statistical data output

Set the configuration file as follows:
Include "common. conf"
Logfile = "/home/Apache/logs/access_log. % YYYY-24 % MM-24 % DD-24"
Sitedomain = "www.chedong.com"
Hostaliases = "chedong.com"
Defaultfile = "index.html"
Dirdata = "/home/Apache/cgi-bin/AWStats/data /"

Summary: Introduction to AWStats and some configuration improvements. We are glad to see that at the beginning of awstats 6.3, Chinese users basically only need to enable loadplugin = "decodeutfkeys" in the configuration file, so there is basically no statistical problem with the Chinese search engine, # minor Chinese search engines 'Baidu \. com ', 'search \. sina \. com ', 'search \. sohu \. com ', the three search engines. Includes patches defined for major search engines and spider in China (unpack and overwrite the originalProgramDirectory)

The log statistics system plays an important role in site user behavior analysis, especially for keyword access statistics from search engines: it is a very effective data source for user behavior analysis. With the development of the Internet over the years, Web Log statistics tools have become more and more mature and feature-rich. Many of them are openSource codeAWStats is an outstanding one.

AWStats: advanced web statistics

AWStats is a fast-growing Perl-based web log analysis tool on SourceForge. Compared with another excellent Open SourceCodeThe log analysis tool Webalizer and AWStats has the following advantages:

1. User-friendly: You can directly call the corresponding language interface (Simplified Chinese Version) based on your browser)
   Reference output sample: http://www.chedong.com/cgi-bin/awstats/awstats.pl? Config = chedong
2. Perl-based: the system can run on GNU/Linux or Windows (after ActivePerl is installed), and the log analysis directly supports the Apache format (combined) and IIS format (need to be modified ). Although Webalizer has a Windows platform version, it lacks maintenance;
   AWStats allows you to use a system to collect statistics on different Web servers on your website: GNU/Linux/Apache and Windows/IIS servers.
3. High Efficiency: the AWStats output statistics project is much richer than Webalizer, and the speed can still reach about 1/3 of Webalizer. This speed is sufficient for websites with millions of daily visits;
4. Convenient configuration/customization: The system provides flexible but reasonable configuration rules by default. The default configuration to be modified cannot exceed 3 or 4 items to start running, there are also many plug-ins for Modification and extension;
5. AWStats designers are designed for precise "Human visits", so robot access to many search engines is filtered out, therefore, the statistics may be lower than those of other log statistics tools. Access from within the company can also be filtered out through IP address filtering settings.
6. Provides many extended Parameter Statistics functions: using the extraxxxx series configuration to generate Parameter Analysis for specific applications is very useful for product analysis.

For more information about Webalizer and analog, see:
Http://awstats.sourceforge.net/#COMPARISON

AWStats installation memo

The AWStats running mode is as follows:

1. Log Analysis: after running, the log statistics are archived into an AWStats database (plain text;
2. Then there is the output in two forms:
   • One is to read the statistical result database output through the CGI program;
   • One is to run the background script to export the output to a static file;

The following are two log statistics examples for a single site:
One is CGI-based output on GNU/Linux,
One is static page-based export on Windows 2000

Download/install

Http://sourceforge.net/projects/awstats/ download installation package:

GNU/Linux: Tar zxf awstats-version.tgz
By default, AWStats scripts and static files are stored in the wwwroot Directory: deploy AWStats. pl programs to/home/Apache/cgi-bin/AWStats /.
MV AWStats-version/cgi-bin/path/to/Apache/cgi-bin/AWStats
# Copy the icons and other file directories to the HTML file publishing directory of the Web:/home/Apache/htdocs/
More batch update scripts can be stored in the cgi-bin/AWStats/directory under the tools directory,

Windows 2000: Run in Script Mode on the background. Unpack the package and move it to the D: \ AWStats directory.
Copy the icon directory to the publishing directory of IIS: inetpub/icon

Data Source log format and day-based truncation rules

1. For Apache: Set the log format to combined. to truncate logs, you must install the cronolog tool to truncate logs by day:
   Customlog "|/usr/local/sbin/cronolog/path/to/Apache/logs/access_log. % Y % m % d" combined
   For example: logs/access_log.20030326
   Logs are compressed. You can use gzip-D
2. For IIS: by default, there are good daily log truncation rules, but the IIS log format is not suitable for AWStats statistics,
   Therefore, it is best to remove all log fields directly and set them strictly according to the following list:
   • Date
   • Time
   • Customer IP address c-IP
   • User name CS-Username
   • Method CS-Method
   • Uri resource CS-Uri-stem
   • Protocol Status SC-status
   • Number of sent bytes SC-bytes
   • Protocol version CS-version
   • User Agent CS (User-Agent)
   • Refer to CS (Referer)

   Compared with IIS default settings:
   There are:

   • Server IP Address
   • Server Port
   • URI query

   Added:

   • Sent bytes
   • Protocol version
   • Reference

Configuration File naming rules: AWStats. sitename. conf

AWStats. pl automatically calls the configuration file AWStats. sitename. conf of the corresponding Site Based on the site name.
For example, running./AWStats. pl-Config = chedong calls the AWStats. chedong. conf configuration file in the same directory;
If-config is not specified, AWStats. conf or/etc/AWStats. conf in the current directory will be used as the default configuration file.
So it is best to rename the default AWStats. model. conf to AWStats. yoursite. conf; for example: AWStats. chedong. conf,

For statistics on multiple sites, the AWStats configuration file inclusion function is still very useful. We can put General configurations in a document and then use it (Versions later than 5.4 will be supported) the include configuration includes the general configuration in the header of each specific configuration file, and then overwrites the corresponding attributes in the general configuration with other configurations, such:
Include = "common. conf"
Logfile = "/path/to/BBS/access_log"
Sitename = "bbs.chedong.com"

Minimal configuration file modification: logfile sitedomain logformat

For Statistics of Apache logs on GNU/Linux, you only need to modify the two options: logfile sitedomain.

1. GNU/Linux logfile = "/path/to/Apache/logs/access_log. % YYYY-24 % MM-24 % DD-24"
   Windows 2000 logfile = "D: \ iis_logs \ w3sv3 \ ex % YY-24 % MM-24 % DD-24.log"
   This configuration indicates the log file name spelled out by the year, month, and date 24 hours ago;
2. Sitedomain = "www.chedong.com"
   The site name. The default value is null. If it is null, AWStats will refuse to run;
3. You need to modify one more log for IIS statistics:
   Logformat = 2
   The default value is 1: Apache log, and 2 is IIS log.

Other precautions:
Awstats if the SWF file is not filtered, The. SWF file will be converted to pageview. Therefore, if the SWF file on the site is primarily an advertisement, it is best to filter it out:

Log Analysis

./AWStats. pl-Update-Config = sitename-lang = Cn
For example:./AWStats. pl-Update-Config =Chedong
Automatically calls AWStats.Chedong. Conf configuration file

Statistical Output

GNU/Linux http: // localhost/cgi-bin/AWStats. pl? Config = chedong
Windows 2000 http: // localhost/AWStats/awstats.chedong.html

Automatic Running of log statistics

On GNU/Linux: crontab-E: Run at 08:10 every day
# Update AWStats
10 8 *** (CD/path/to/Apache/cgi-bin/AWStats/;./AWStats. pl-Update-Config = chedong)

On Windows 2000: set to run at 08:10 every day
D: \ Perl \ bin \ perl.exe D: \ AWStats \ tools \ awstats_buildstaticpages.pl-Update-Config = chedong-lang = cn-Dir = C: \ Inetpub \ AWStats \-awstatsprog = D: \ AWStats \ wwwroot \ cgi-bin \ AWStats. PL

Multi-site log statistics

AWStats comes with a batch processing tool: Tools/awstats_updateall.pl, which allows you to traverse all configuration files in a directory in batches and run the unified computing. Therefore, the remaining work is mainly due to log synchronization issues.

For multiple sites, many configuration options are repeated. If each configuration file is modified and maintained, it will be very troublesome. AWStats provides the configuration file inclusion function starting from version 5.4, so we can configure a general configuration, such as: Common. conf

The configuration of other sites is as follows: You can overwrite the configuration that is inconsistent with the default one using the following options.
AWStats. bbs. chedong. conf
Include "chedong. Common. conf"
Logfile "/path/to/bbs_log"
Sitename "bbs.chedong.com"

AWStats. www. chedong. conf
Include "chedong. Common. conf"
Logfile "/path/to/www_log"
Sitename "www.chedong.com"
Hostaliases = "chedong.com"

Statistical indicator description

• Visitor: collect statistics based on unique IP addresses of visitors. One IP address represents one visitor;
• Number of visits: A visitor may visit multiple times within one day (for example, once in the morning or once in the afternoon), so within a certain period of time (for example, 1 hour ), number of non-repeated IP addresses, visitor visits;
• Webpage count: the total number of page visits, excluding images, CSS, and JavaScript files. However, if a page uses multiple frames, each frame is regarded as a page request;
• Number of Files: Total number of file requests from the browser client, including images, CSS, JavaScript, etc. The user requests a page, if the page contains images, etc, therefore, the server will send multiple file requests, and the number of files is generally far greater than the number of files;
• Byte: the total data traffic sent to the client;
• Data from Referer: the reference (Referer) field in the log records the address before accessing the corresponding webpage. Therefore, if you click to enter the website through the search results of the search engine, the log contains the user's query address in the corresponding search engine. This address can be parsed to extract the keywords used by the user's query:
  For example:
  15:43:58 123.123.123.123.123-Get/index.html 200 192 HTTP/1.1 Mozilla/4.0 + (compatible; + MSIE + 5.01; + windows + nt + 5.0) http://www.google.com/search? Q = chedong
  AWStats has complete functions in key phrases and keyword statistics of search engines: it can identify more than types of Web Crawlers around the world, it can also identify most mainstream international search engines and local language search engines in many regions.

Hacking AWStats

Install GIS-based plug-ins:

Geoip and GEO: ipfree (AWStats 5.5 +)
Geoip and GEO: ipfree are free of charge. They are more accurate and faster than those obtained through reverse DNS resolution. Geoip APIs are free of charge, and the default database is free of charge. The charge is its Data Update Service. GEO: ipfree not only makes code public, but also makes library data public.

Geoip installation:
Download the C library first: After geoip C is unwrapped
%./Configure; make
# Make install

Download the Perl Library: geoip Perl.
% Perl makefile. pl; make
# Make install

GEO: ipfree installation:
Download GEO: ipfree package
% Perl makefile
% Make
# Make install

Configuration: Enable geoip-related plug-ins in the configuration file:

Loadplugin = "geoip geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geoip.dat"
Loadplugin = "geoip_city_maxmind geoip_standard/home/Apache/chedong.com/cgi-bin/awstats/geolitecity.dat"

Maxmind currently provides geoip and geoipcitylite data packets for free: You can regularly download data from the following addresses every month:

Http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
Http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz