AWStats powerful log analysis tool

Preface: The company used awstats for web server traffic statistics. After looking for 51cto, it found that there were very few articles. So I wrote this article to record my own configuration process, we also hope to help our friends in need. powerful log analysis tools AWStats home page http://awstats.sourceforge.net Note: This article basically according to the official documentation step by step, in the installation process, refer to the relevant documentation made necessary changes. environment: redhat 9 Apache/2.0.54 source code installation. The path is/usr/local/apache2 Host IP address 192.168.0.111domain name s1.domain1.com. the awstats-6.5.tar.gz function is used to collect s1.domain1.com access information, there are many features available to view AWStats on the web page. For details, click the official website above. Here I mainly use it to analyze the logs of the apache server. the general principle of awstats is as follows: awstats provides a series of perl scripts to implement functions such as service configuration, log reading, and report generation. the specific implementation process of the function is: first, of course, apache records the access information to the log, AWStats reads the logs every time it executes the update, and analyzes the log data, store the results in the database (this database comes with awstats and does not require third-party software support .), finally, AWStats provides a cgi program to display the statistical data in the database through the web page. first, check the current apache configuration, vi httpd. conf<VirtualHost *: 80> ServerAdminyahoon@xxx.comDocumentRoot/var/www/html/s1ServerNames1. domain1.comErrorLog logs/s1_web-error_logTransferLog logs/s1_web-access_log </VirtualHost>Access s1.domain1.com logs to/usr/local/apache2/logs/s1_web-access_log the installation procedure is as follows decompress the package to/usr/local/awstats the first step is to generate a configuration file for the website, in fact, there are many ways to recommend using scripts according to the instructions in the official documentation, and this is also a relatively simple method. go to/usr/local/awstats/tools. There are many perl scripts with different functions in this directory. run [root @ server1 tools] # perl awstats_configure.pl to enter an interactive configuration program. Some questions may be answered during the process. Of course, these can be modified in future configuration files. I will list all the content and annotations as follows:----- AWStats awstats_configure 1.0 (build 1.6) (c) Laurent Destailleur ----- This tool will help you to configure AWStats to analyze statistics forone web server. you can try to use it to let it do all that is possiblein AWStats setup, however following the step by step manual setupdocumentation (docs/index.html) is often a better idea. above all if:-You are not an administrator user,-You want Analyze downloaded log files without web server,-You want to analyze mail or ftp log files instead of web log files,-You need to analyze load balanced servers log files, -You want to 'understamp' all possible ways to use AWStats... Read the AWStats documentation (docs/index.html). --- à Running OS detected: Linux, BSD or UnixDetected my current operating system-----> Check for web server installFound Web server Apache config file '/usr/local/apache2/conf/httpd. conf'Detected my current apache configuration file-----> Check and complete web server config file '/usr/local/apache2/conf/httpd. conf 'Warning: You Apache config file contains ctictives to write 'common' log filesThis means that some features can't work (OS, browsers and keywords detection ). do you want me to setup Apache to write 'combined' log files [y/N]? YWe recommend that you use combined logs for AWStats. apache uses the common format by default. Therefore, you are prompted to modify the apache configuration file and change the log format to combined. Of course, the answer is yes.Add 'Alias/awstatsclasses "/usr/local/awstats/wwwroot/classes/" 'add' Alias/awstatscss "/usr/local/awstats/wwwroot/css/" 'add' alias/awstatsicons "/usr/local/awstats/wwwroot/icon/" 'add' ScriptAlias/awstats/"/usr/local/awstats/wwwroot/cgi-bin/" 'Add '<Directory>' ctictiveawstats directives added to Apache config file.Add the above content to the apache configuration file, which contains some directory and alias settings. This part is generally added to the end of the file.-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats. model. conf' File awstats. model. conf updated.Template file updated-----> Need to create a new config file? Do you want me to build a new AWStats config/profilefile (required if first install) [y/N]? YWhether to create a new configuration file, select y-----> Define config file name to createWhat is the name of your web site or profile analysis? Example: [url] www.mysite.com [/url] Example: demoYour web site, virtual server or profile name: Ø s1 .domain1.comEnter the Domain Name of the website in this region-----> Define config file pathIn which directory do you plan to store your config file (s )? Default:/etc/awstatsDirectory path to store config file (s) (Enter for default):> press EnterThe path where the generated configuration file is stored. The default path is/etc/awstats. If you do not modify the path, press Enter.-----> Create config file '/etc/awstats/awstats.s1.domain1.com. conf' Config file/etc/awstats/awstats.s1.domain1.com. conf created.The file has been created, and the path is/etc/awstats/awstats.s1.domain1.com. conf'-----> Restart Web server with '/sbin/service httpd restart' Stopping httpd: [OK] Starting httpd: cocould not determine the server's fully qualified domain name, using 127.0.0.1 for ServerName [OK]Because httpd is modified. conf, so it will automatically restart apache to make the configuration file take effect, but there is a problem here, because my apache does not use the built-in system, but the source code is installed, it has not been added to the service, so the startup is successful, but it is not what I need. it's easy. I will close it and enable my own.-----> Add update process inside a schedulerSorry, configure. pl does not support automatic add to cron yet. you can do it manually by adding the following command to your cron:/usr/local/awstats/wwwroot/cgi-bin/awstats. pl-update-config = s1.domain1. comOr if you have several config files and prefer having only one command:/usr/local/awstats/tools/awstats_updateall.pl nowPress ENTER to continue...To obtain new information, you must update the database, that is, re-read the log. analyze the newly added parts in the log extraction to save the updated data to the database. it is best to make it run regularly. this section reminds you that to implement this function, you need to manually write any row of the above two rows in bold to crontab for regular execution. the first sentence is to update s1.domain1.com, and the second sentence is to update all sites when there are multiple sites. I will not perform this operation here. press enter.A simple config file has been created:/etc/awstats/awstats.s1.domain1.com. confYou shoshould have a look inside to check and change manually main parameters. you can then manually update your statistics for 's1 .domain1.com 'with command:> perl awstats. pl-update-config = s1.domain1. comYou can also read your statistics for 's1 .domain1.com 'with URL: Ø [url] http: // localhost/awstats. pl? Config = s1.domain1.com [/url] Press ENTER to finish...Press enter to end the installation. this section is clearly written. The next task is to check the configuration file and execute perl awstats. pl-update-config = s1.domain1.com to update the database through [url] http: // localhost/awstats. pl? Config = s1.domain1.com [/url] To View statistics. After this script is executed, our httpd. conf is updated. The following is the main part of the script.Change CustomLog logs/access_log common to CustomLog logs/access_log combined and add # Directives to allow use of AWStats as a CGIAlias/awstatsclasses "/usr/local/awstats/wwwroot /classes/"Alias/awstatscss"/usr/local/awstats/wwwroot/css/"Alias/awstatsicons"/usr/local/awstats/wwwroot/icon/"ScriptAlias/awstats/ "/usr/local/awstats/wwwroot/cgi-bin/" # This is to permit URL access to scripts/files in AWStats directory. <Directory "/usr/local/awstats/wwwroot"> Options NoneAllowOverride NoneOrder allow, denyAllow from all </Directory>Before modification, stop apache started by the script, start the correct apache [root @ server1 tools] # service httpd stop [root @ server1 tools] #/usr/local/apache2/bin/apachectl start and access the website s1.domain1.com, this access should be recorded in the log. access [url] http: // 192.168.0.111t/awstats. pl? Config = s1.domain1.com [/url], promptForbiddenYou don't have permission to access/awstats. pl on this serverIt is clear that it is a permission issue. Execute chmod-R 777 awstats under/usr/local (it should be 755, because I only want to test it) check the key items in the configuration file vi/etc/awstats/awstats.s1.domain1.com. check the conf file and make the following changes:# LogFile = "/var/log/httpd/mylog. log "LogFile ="/usr/local/apache2/logs/s1_web-access_log "indicates that the log LogType = W indicates that the web log LogFormat = 1 indicates that the log format is combinedSiteDomain =" s1.domain1.com the variable "HostAliases =" s1.domain1.com [url] www.s1.domain1.com [/url] 127.0.0.1 localhost "indicates the alias of this domain. this statement is automatically generated when multiple domain names correspond to the same website. I didn't use it here, so I didn't change it.Update the database cd/usr/local/awstats/wwwroot/cgi-binperl awstats. pl-config = s1.domain1.com-update. the following error is returned:Error: AWStats database directory defined in config file by 'dirdata' parameter (/var/lib/awstats) does not exist or is not writable. setup ('/etc/awstats/awstats.s1.domain1.com. conf 'file, web server or permissions) may be wrong. check config file, permissions and AWStats documentation (in 'docs' directory ).Simultaneously access [url] http: // 192.168.0.111t/awstats. pl? Config = s1.domain1.com [/url] promptError: AWStats database directory defined in config file by 'dirdata' parameter (/var/lib/awstats) does not exist or is not writable. setup ('/etc/awstats/awstats.s1.domain1.com. conf 'file, web server or permissions) may be wrong. check config file, permissions and AWStats documentation (in 'docs' directory ).Obviously, the/var/lib/awstats directory does not exist. It should be noted that this path is set by the configuration file/etc/awstats/awstats.s1.domain1.com. the DirData parameter in conf determines where the database is stored. I have not modified it, so it is still the default value DirData = "/var/lib/awstats". Since the prompt does not exist, now we will create it cd/var/libmkdir awstats for database updates [root @ server1 cgi-bin] # perl awstats. pl-config = s1.domain1.com-update information is as follows:Update for config "/etc/awstats/awstats.s1.domain1.com. conf "With data in log file"/usr/local/apache2/logs/s1_web-access_log "... phase 1: First bypass old records, searching new record... searching new records from beginning of log file... jumped lines in file: 0 Parsed lines in file: 1 Total number of lines processed (apache logs represent a record) Found 0 dropped records, Found 1 lost upted records, A corrupt record Found 0 old records and Found 0 new qualified records is Found.View the log Content [root @ server1 cgi-bin] # less/usr/local/apache2/logs/s1_web-access_log192.168.0.28--[30/Aug/2007: 10: 19: 59 + 0800] "GET/HTTP/1.1" 200 23 access [url] http: // 192.168.0.111/awstats. pl? Config = s1.domain1.com [/url] It is strange to view the statistics. Why? Open httpd. conf does not change the part of the VM, so the log type recorded has not changed. You need to set the Log Type separately. serverAdminyahoon@xxx.comDocumentRoot/var/www/html/s1ServerName s1.domain1. comLogFormat "% h % l % u % t \" % r \ "%> s % B \" % {Referer} I \ "\" % {User-Agent} I \ "" combined1CustomLog logs/s1_web-access_log combined1ErrorLog logs/s1_web-error_log </VirtualHost> You Can See That I named this log format combine1 here, it's just a name. restart apache after modification to clear the log, and then access s1.domain1.com, the generated log is as follows [root @ server1 cgi-bin] # less/usr/local/apache2/logs/s1_web-access_log192.168.0.28--[30/Aug/2007: 13: 04: 05 + 0800] "GET/HTTP/1.1" 304-"-" "Mozilla/4.0 (compatible; MSIE 6.0; windows NT 5.1; SV1; TencentTraveler) "192.168.0.28--[30/Aug/2007: 13: 05: 30 + 0800] "GET/HTTP/1.1" 304-"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler) "192.168.0.28--[30/Aug/2007: 13: 05: 30 + 0800]" GET/HTTP/1.1 "304-"-"" Mozilla/4.0 (compatible; MSIE 6.0 ; Windows NT 5.1; SV1; TencentTraveler) "192.168.0.28--[30/Aug/2007: 13: 05: 30 + 0800] "GET/HTTP/1.1" 304-"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler) "192.168.0.28--[30/Aug/2007: 13: 05: 31 + 0800]" GET/HTTP/1.1 "304-"-"" Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler) "192.168.0.28--[30/Aug/2007: 13: 05: 31 + 0800]" GET/HTTP/1.1 "304 -"-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler )"......Update the database again [root @ server1 cgi-bin] # perl awstats. pl-config = s1.domain1.com-updateUpdate for config "/etc/awstats/awstats.s1.domain1.com. conf "With data in log file"/usr/local/apache2/logs/s1_web-access_log "... phase 1: First bypass old records, searching new record... direct access to last remembered record has fallen on another record. so searching new records from beginning of log file... phase 2: Now process new records (Flush history on disk after 20000 hosts )... jumped lines in file: 0 Parsed lines in file: 6 Found 0 dropped records, Found 0 reset upted records, Found 0 old records, Found 6 new qualified records.Access http: // 192.168.0.111/awstats. pl? Config = s1.domain1.com the last thing to note is that the database must be updated to reflect the latest information, which has been mentioned many times in this article. that is to say, you may have accessed it multiple times, but the statistics page does not change. this is because these logs have not been updated to the database. the simple method is regular execution.

Of course, it is only about the basic installation and use of the software, and its use is far from the end. in particular, it involves apache Log related knowledge, such as rotate log loop and load balancing log merging. I am also new to contact you. I hope you can give me more valuable comments.

Add password verification for awstats log view

Vim/usr/local/apache2/conf/httpd. conf

<Files "awstats. pl">
AuthType Basic
AuthName "awstat access"
AuthUserFile "/usr/local/awstats/htpasswd. users"
Require valid-user
</Files>

Then

/Usr/local/apache2/bin/htpasswd-c/etc/awstats/htpasswd admin

New password:
Re-type new password:
Adding password for user admin

/Usr/local/apache2/bin/apachectl stop

/Usr/local/apache2/bin/apachectl start

You can.