Awstats website traffic statistics _ MySQL

Awstats website traffic statistics, mainly used

Install, Setup and Use AWStats
AWStats common use is made in 3 steps:

Step 0: The install and setup
Step 1: The build/update of statistics

Step 2: The reading of results

Before starting, check that your Perl version is at least 5.005 _ 03 (or higher) by running the perl-v command. if not, you can install a recent Perl interpreter from ActivePerl (Win32) or Perl.com (Unix/Linux/Other ).
-->

Step 0: Install and Setup with awstats_configure.pl

A) With Apache or compatible server (on Unix/Linux, Windows, MacOS ...)

* Step 0-1:
(If you use a package provided with a Linux distribution or Windows installer, action done in step 0-1 might have already be done, if you don't know you, do it again)

After downloading and extracting the AWStats package, you should run the awstats_configure.pl script to do several setup actions. you will find it in the AWStats tools directory (If using the windows installer, the script is automatically launched ):
Perl awstats_configure.pl

This is what the script do/ask (you can do all those steps manually instead of running awstats_configure.pl if you prefer ):

A) awstats_configure.pl try to find your Apache web server config file (ask path if not found) and check inside your server log format configuration. if you use a common log, awstats_configure.pl will suggest to change it to have NCSA combined/XLF/ELF log format (you can use your own log format but this predefined logformat is often the best choice and make setup easier ).
If you answer yes, awstats_configure.pl will modify your httpd. conf to change all the following ctictives:
CustomLog/yourlogpath/yourlogfile common
Into
CustomLog/yourlogpath/yourlogfile combined

See your Apache manual for more information on this directive.
To be sure the log format change is valid, later you can stop Apache, remove all old log files, restart Apache and go to your homepage. this is an example of records you shocould get then in your new log file:
62.161.78.75--[dd/mmm/yyyy: hh: mm: ss + 0000] "GET/HTTP/1.1" 200 1234 "http://www.from.com/from.html" "Mozilla/4.0 (compatible; MSIE 5.01; windows NT 5.0 )"

B) Then, awstats_configure.pl will add, if not already present, the following directives to your Apache config file (note that the "/usr/local/awstats/wwwroot" path might differ according to your distribution or OS:

#
# Directives to add to your Apache conf file to allow use of AWStats as a CGI.
# Note that path "/usr/local/awstats/" must reflect your AWStats install path.
#
Alias/awstatsclasses "/usr/local/awstats/wwwroot/classes /"
Alias/awstatscss "/usr/local/awstats/wwwroot/css /"
Alias/awstatsicons "/usr/local/awstats/wwwroot/icon /"
ScriptAlias/awstats/"/usr/local/awstats/wwwroot/cgi-bin /"
#
# This is to permit URL access to scripts/files in AWStats directory.
#

Options None
AllowOverride None
Order allow, deny
Allow from all

C) awstats_configure.pl restart Apache to apply the changes made in A and B (if changes were made ).

D) awstats_configure.pl will ask you a name for a config profile file. Enter here the name of your web server or any analysis profile name, for example myvirtualhostname.
So awstats_configure.pl will copy the file awstats. model. conf file into a new file named awstats. myvirtualhostname. conf. you can use the value of your choice instead of "myvirtualhostname ". this new file is stored:
-For Linux/BSD/Unix users:/etc/awstats.
-For Mac OS x, Windows and other OS: Same directory than awstats. pl (so cgi-bin ).

E) awstats_configure.pl ends.

* Step 0-2:
Once a config file has been automatically created (by awstats_configure.pl, by your package installer or just by a manual copy of awstats. model. conf), it's important to edit it manually and change the "main parameters" to match all your needs:

This is for example the parameters you shoshould check seriously:
-Check/Change LogFile value with full path of your server log file (You can also use a relative path from your awstats. pl directory, but full path avoid errors ).
-Check/Change LogType value with "W" for analyzing web server log files, "S" for a streaming server log file, "M" for mail log files, "F" for ftp log files.
-Check if LogFormat has the value "1" (it means "NCSA apache combined/ELF/XLF log format ") or use a personalized log format if you don't use combined log format.
-Edit SiteDomain parameter with the main domain name or the intranet web server name used to reach the web site to analyze (Example: www.mydomain.com ). if you have several possible name for same site, add list in HostAlias parameter.
-You can also change other parameters if you want.

Step 0 (Install and Setup) is finished. You can jump to the Build/Update Statistics section.

B) With IIS server

* Step 0-1:

Configure IIS to log in "Extended W3C log format" (You can still use your own log format but setup is easier if made like suggested ). so, for this, start the IIS Snap-in, select the web site and look at its Properties. choose W3C Extended Log Format, then Properties, then the Tab Extended Properties and uncheck everything under Extended Properties. once they are all unchecked, check all following fields:
Date
Time
C-ip
Cs-username
Cs-method
Cs-uri-stem
Cs-uri-query
SC-status
SC-bytes
Cs-version
Cs (User-Agent)
Cs (Referer)
To be sure the log format change is valid, you must stop IIS, remove all old log files, restart IIS and go to your homepage. this is an example of records you shocould get then in the new log file:
14:14:14 62.161.78.73-GET/200 1234 HTTP/1.1 Mozilla/4.0 + (compatible; + MSIE + 5.01; + Windows + NT + 5.0) http://www.from.com/from.htm

* Step 0-2:

Copy the contents of the provided cgi-bin folder from your hard drive to your server's cgi-bin directory (this includes des awstats. pl, awstats. model. conf, and the lang, lib and plugins sub-directories ).

* Step 0-3:

Move AWStats icon sub-directories and its content into a directory readable by your web server, for example C:/yourwwwroot/icon.

* Step 0-4:

Create a config file by copying awstats. model. conf file into a new file named awstats. myvirtualhostname. conf. you can use the value of your choice instead of "myvirtualhostname ". this new file must be stored in
-Same directory than awstats. pl (so cgi-bin)

* Step 0-5:

Edit this new config file with your own setup:
-Change LogFile value with full path of your web server log file (You can also use a relative path from your awstats. pl directory ).
-Change LogType value with "W" for analyzing web server log files, "S" for a streaming server log file, "M" for mail log files, "F" for ftp log files, "O" otherwise.
-Change LogFormat to a value with same field's name defined in step 0-1:
LogFormat = "date time c-ip cs-username cs-method cs-uri-stem cs-uri-query SC-status SC-bytes cs-version cs (User-Agent) cs (Referer )"
-Change DirIcons parameter to reflect relative path of icon directory.
-Edit SiteDomain parameter with the main domain name or the intranet web server name used to reach the web site to analyze (Example: www.mydomain.com ).
-Set AllowToUpdateStatsFromBrowser parameter to 1 if you don't have command line access and have only cgi access.
-You can change other parameters if you want.

Step 0 (Install and Setup) is finished. You can jump to the Build/Update Statistics section.

C) With other web servers

Setup process is similar to setup for Apache or IIS.
Use LogFormat to value "3" if you have WebStar native log format, use a personalized LogFormat if your log format is other.

Step 1: Build/Update Statistics

* Step 1-1:

The first analyze/update of statistics shoshould be made the first time manually from the command line since first time, process may be long and it's easier to solve problems (if you don't have Command Line access, just go to Step 1-2 ). the AWStats update command line is:
Awstats. pl-config = myvirtualhostname-update

AWStats will read the config file awstats. myvirtualhostname. conf (or if not found, awstats. conf) and create/update its database with all summary information issued from analyzed log file.
AWStats database files are saved in directory defined by DirData parameter in config file.
When update is finished, you shoshould get on screen a result like this:

Update for config "/etc/awstats. myvirtualhostname. conf"
With data in log file "/pathtoyourlog/yourlog. log "...
Phase 1: First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2: Now process new records (Flush history on disk after 20000 hosts )...
Jumped lines in file: 0
Parsed lines in file: 225730
Found 122 dropped records,
Found 87 successfully upted records,
Found 0 old records,
Found 225521 new qualifed records.

Dropped records are records discarded because they were not user HTTP request or requests were not qualified by AWStats filters (See SkipHosts, keys, SkipFiles, OnlyHosts, parameters and OnlyFiles parameters ). if you want to see which lines were dropped, you can add the-showdropped option on command line.
Specified upted records are records that does not match log format defined by "LogFormat" parameter in AWStats config/domain file. with all webservers you can experience a little bit into upted records (<5%) even when everythings work correctly. this can be the result of several reasons: Web server internal bugs, bad requests made by buggy browsers, web server brutal stop...
If all your lines are upted and LogFormat parameter in AWStats config/domain file is correct, it may be the log format setup in your web server that is wrong. don't forget that your LogFormat parameter in AWStats config/domain file MUST match the log file format you analyze.
If you want to see which lines are already upted, you can add the-showupted option on command line.
Old records are simply records that were already processed by a previous update process. you understood that it's not necessary to purge your log file after each update process even if it's highly recommended to do it as often as possible.
New records are records in your log file that were successfully used to build/update statistics.

Note: A log analysis process is slow (one second for each 4500 lines of your logfile with Athlon 1 Ghz, plus DNS resolution time for each different IP address in your logfile if DNSLookup is set to 1 and not already done in your log file ).
See Benchmark page for more accurate information.

* Step 1-2:

AWStats statistics can be updated from a browser, to provide real-time statistics, by clicking the "Update now" link that appears when AWStats is used as a CGI (Next section 'read Statistics dynamically 'gives you URL to use for that ).
Warning !!
To enable this link, your parameter AllowToUpdateStatsFromBrowser must be set to 1 in your config file (Link is not enabled by default ).

Then, using on-line update does not prevent you from running the update process from a scheduler frequently (command is same than update of first process ).
For this, you have two choices:
-Include the update command in your logrotate process. See FAQ-COM120 for this.
-Or add instructions in your crontab (Unix/Linux) or your task scheduler (for Windows), to launch frequently the Awstats update process. See FAQ-COM130 for this.

See AWStats Benchmark page for recommanded update/logrotate frequency.

Step 2: Read Statistics

To see results of analyze, you have several solutions depending on your security policy.

* First solution is to build the main reports, in a static HTML page, from the command line, like this (jump to second solution if you have only cgi access ):
Perl awstats. pl-config = myvirtualhostname-output-staticlinks> awstats.myvirtualhostname.html

You can use all other output options (each of them give you another report). This is how to use all other possible output options (1 ):
Perl awstats. pl-config = myvirtualhostname-output = alldomains-staticlinks> awstats.myvirtualhostname.alldomains.html
Perl awstats. pl-config = myvirtualhostname-output = allhosts-staticlinks> awstats.myvirtualhostname.allhosts.html
Perl awstats. pl-config = myvirtualhostname-output = lasthosts-staticlinks> awstats.myvirtualhostname.lasthosts.html
Perl awstats. pl-config = myvirtualhostname-output = unknownip-staticlinks> awstats.myvirtualhostname.unknownip.html
Perl awstats. pl-config = myvirtualhostname-output = alllogins-staticlinks> awstats.myvirtualhostname.alllogins.html
Perl awstats. pl-config = myvirtualhostname-output = lastlogins-staticlinks> awstats.myvirtualhostname.lastlogins.html
Perl awstats. pl-config = myvirtualhostname-output = allrobots-staticlinks> awstats.myvirtualhostname.allrobots.html
Perl awstats. pl-config = myvirtualhostname-output = lastrobots-staticlinks> awstats.myvirtualhostname.lastrobots.html
Perl awstats. pl-config = myvirtualhostname-output = urldetail-staticlinks> awstats.myvirtualhostname.urldetail.html
Perl awstats. pl-config = myvirtualhostname-output = urlentry-staticlinks> awstats.myvirtualhostname.urlentry.html
Perl awstats. pl-config = myvirtualhostname-output = urlexit-staticlinks> awstats.myvirtualhostname.urlexit.html
Perl awstats. pl-config = myvirtualhostname-output = browserdetail-staticlinks> awstats.myvirtualhostname.browserdetail.html
Perl awstats. pl-config = myvirtualhostname-output = osdetail-staticlinks> awstats.myvirtualhostname.osdetail.html
Perl awstats. pl-config = myvirtualhostname-output = unknownbrowser-staticlinks> awstats.myvirtualhostname.unknownbrowser.html
Perl awstats. pl-config = myvirtualhostname-output = unknownos-staticlinks> awstats.myvirtualhostname.unknownos.html
Perl awstats. pl-config = myvirtualhostname-output = refererse-staticlinks> awstats.myvirtualhostname.refererse.html
Perl awstats. pl-config = myvirtualhostname-output = refererpages-staticlinks> awstats.myvirtualhostname.refererpages.html
Perl awstats. pl-config = myvirtualhostname-output = keyphrases-staticlinks> awstats.myvirtualhostname.keyphrases.html
Perl awstats. pl-config = myvirtualhostname-output = keywords-staticlinks> awstats.myvirtualhostname.keywords.html
Perl awstats. pl-config = myvirtualhostname-output = errors404-staticlinks> awstats.myvirtualhostname.errors404.html

Note (1): If you prefer, you can use awstats_buildstaticpages tool to build all those pages in one command or to generate PDF files.

Note (2): You can also add a filter on the following output reports: urldetail, urlentry, urlexit, allhosts, refererpages.
Filter can be a regexp on the full key you want awstats to present information about and you must use it after the output parameter separated by ":".
For example, to output the urldetail report, with a filter on all pages that contains/news, you can use the following command line:
Perl awstats. pl-config = myvirtualhostname-output = urldetail:/news-staticlinks> awstats.myvirtualhostname.urldetailwithfilter.html

Note (3): If you want to build a report for a participant month, add options-month = MM-year = YYYY.
To build a report for full year (warning: This may use a lot of memory and CPU), add options-month = all-year = YYYY.

* Second solution is to view dynamically your statistics from a browser. For this, use URL:
Http://www.myserver.mydomain/awstats/awstats.pl? Config = myvirtualhostname
Where myvirtualhostname is used to know which config file to use (AWStats will use awstats. myvirtualhostname. conf file ).

Note (1): All output command line options (except-staticlinks) are still available when using AWStats as a browser.
Just use them as URL parameters like this example http://www.myserver.mydomain/awstats/awstats.pl? Month = MM & year = YYYY & output = unknownos

Note (2): If AllowToUpdateStatsFromBrowser parameter is set to 1 in AWStats config/domain file, you will also be able to run the update process from your browser. just click on link "Update now ".