Use awstats to analyze Nginx access logs

1. configuring Nginx Automatic Log cutting is different from ApacheHTTPServer (Apache). Apache can redirect the log output through pipelines to enable automatic log cutting. Nginx in the current version has not been the same as Apache, using parameters such as % YY to create logs in batches by date, but

1. Configure Nginx to automatically cut logs
Different from Apache HTTP Server (Apache), Apache can redirect log output through pipelines to perform Automatic Log cutting. In today's version, Nginx has not been able to create logs in batches by date using parameters such as % YY, but by sending a specific signal to the nginx process, nginx can be used to generate a log file again. We can regularly execute a Shell script to switch logs, rename or transfer logs. The specific script is as follows:

Mv/Opt/nginx/logs/access. log/opt/nginx/logs/access _'Date+ % Y % m % D'. log
KillAll-s USR1 nginx # Use the USR1 parameter to notify the Nginx process to switch the log file to save the above script content as the file name is logcron. sh to the custom directory, such as/opt/nginx/sbin/logcron. sh
Use Crontab to automatically execute the script, and create logs by day.

2. install and configure AwStatS
Before installation, make sure that the Perl environment on your server is ready. We also need to make a small modification to the Nginx log format, otherwise awstats will not be able to make statistics.

Vi/opt/nginx/conf/nginx. conf
 
Server {
Listen 80;
Server_name localhost;
....
Log_foRmAt new_log # format alias (Note: if there are multiple virtual hosts, the alias cannot be the same)
\ '$ Remote_aDdR-$ remote_user [$Time_ Local] $ request \'
\ '$ Status $ body_bytes_sent $ http_referer \'
\ '$ Http_user_agent $ http_x_forwardEd_ \';
Access_log logs/access. log new_log; # log generation path
 
}
Download the latest awstats package. Run the downloaded tar packageExtractTo any directory, for example,/usr/local/awstats. Then execute tooLsThe awstats_configure.pl Configuration Wizard in the directory to create a new statistics.

-----> Check for web server install
 
Enter full configFilePath of your Web server.
Example:/etc/Httpd/Httpd. conf
Example:/usr/local/apache2/conf/httpd. conf
Example: c: Program filesapache groupapacheconfhttpd. conf
Config file path (\ 'None \ 'to skip web serverSetUp ):
#> None # Because Nginx is used here, write none and skip. Enter

Your web server config file (s) cocould not be found.
You will needSetupYour web server manually to declare AWStats
Script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/indEx. Html)
 
-----> Update model config file \ '/usr/local/awstats/wwwroot/cgi-bin/awstats. model. conf \'
File awstats. model. conf updated.
 
-----> Need to create a new config file?
Do you want me to build a new AWStats config/profile
File (required if first install) [y/N]?
#> Y # y create a new statistical configuration and press Enter.

-----> Define config file name to create
What is the name of your web site or profile analysis?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
#> Www.key0.cn # website domain name example:
Enter www.key0.cn

-----> Define config file path
InWhichDirectory do you plan to store your config file (s )?
Default:/etc/awstats
Directory path to store config file (s) (Enter for default ):
#> Press enter by default. The following prompt is displayed.

----> Add update process insIdE a scheDuLer
Sorry, configure. pl does notSuPport automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/Usr/local/awstats/wwwroot/cgi-bin/awstats. pl-update-config = www.key0.cn
# Return to thisCommandEnterCrontabRun at specified time
Or if you have several config files and prefer having only one command:
/Usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue... Press ENTER to continue
 
A simple config file has been created:/etc/awstats/awstats.www.key0.cn. conf
# Path of the new configuration file
You shoshould haveLookInside to check and change manually main parameters.
You can then manually update your statistics for \ 'www .key0.cn \ 'with command:
> Perl awstats. pl-update-config = www.key0.cn
You can also build static report pages for \ 'www .key0.cn \ 'with command:
> Perl awstats. pl-output = pagetype-config = www.key0.cn
 
Press ENTER to finish... Press ENTER to complete the wizard, and then modify the statistical configuration of www.key0.cn.
# Vi/etc/awstats/awstats.www.key0.cn. conf
Find the path of the log file for Statistics
LogFile = "/var/log/httpd/mylog. log"
Change
LogFile = "/opt/nginx/logs/access _ % YYYY-0 % MM-0 % DD-0.log
Corresponding to the directory storage structure generated by the above Nginx log cutting program, note that the Awstats format is different from that of Nginx. The order of statistics is as follows:
Logs generated by Nginx-> log cutting-> continue to generate logs by Nginx-> Save cut logs-> submit to Awstats for statistics-> generate results
In this article, the log collected by Awstats is the part that has been cut down. The order can also be adjusted. After the statistics are completed, the switch is performed. However, this may easily result in missing statistics. After the configuration is modified, save and exit. Then we can start to execute it manually.
Run the logcron. sh script to cut Nginx logs.
Then run the Awstats log update program to start statistical analysis.

/Opt/nginx/sbin/logcron. sh
/Usr/local/awstats/wwwroot/cgi-bin/awstats. pl-update-config = www.key0.cn
 
Create/Update database for config/etc/awstats/awstats.www.key0.cn. conf
By AWStats version 6.7 (build 1.892)
From data in log file/opt/nginx/logs/access_20080804.log...
Phase 1: First bypass old records, searching new record...
Direct access after last parSedRecord (after line 450421)
Jumped lines in file: 450421
Found 450421 already parsed records.
Parsed lines in file: 120
Found 0 dropped records,
Found 0 successfully upted records,
Found 0 old records,
Found 120 new qualified records. The above shows that log cutting and Awstats are running correctly. After the statistical analysis is complete, the results are still in the Awstats database. On Apache, you can directly open the Perl program webpage to view statistics. But as mentioned at the beginning of this article, Nginx does not support Perl well, so we need to change the method to use the awstats tool to generate static files for the statistical results. The specific steps are as follows:
First, create a folder under the webroot directory. Example:/data/webroot/awstats
Then let Awstats generate the static page to this directory.

Mkdir/Data/webroot/awstats
 
/Usr/local/awstats/tools/awstats_buildstatiCpAges. pl-update
-Config = www.key0.cn-lang = cn-dir =/data/admin_web/awstats
-Awstatsprog =/usr/local/awstats/wwwroot/cgi-bin/awstats. pl:
/Usr/local/awstats/tools/awstats_buildstaticpages.pl Awstats static page generation tool
-Update-config = www.key0.cn: update Configuration items
-Lang = cn: Chinese
-Dir =/data/admin_web/awstats output directory
-Awstatsprog =/usr/local/awstats/wwwroot/cgi-bin/awstats. pl Awstats log update program path.
Next, configure the directory in nginx. conf. Example: (bold ):

Server {
Listen 80;
Server_name localhost;
 
LoCatIon ~ ^/Web /{
Root/data/web;
Index index.html;
Error_log off;
Charset gb2312;
}
 
Location ~ ^/Awstats/{# html static page Directory
Root/data/webroot/awstats;
Index index.html;
Access_log off;
Error_log off;
Charset gb2312; # It is best to change the default encoding to gb2312 to avoid garbled characters in the browser due to automatic encoding.
}
 
Location ~ ^/Icon/{# icon directory
Root/usr/local/awstats/wwwroot;
Index index.html;
Access_log off;
Error_log off;
Charset gb2312;
}
} View the detailed statistics result in a browser: http: // youhostname/awstats/awstats.www.key0.cn.html
So far, awstats fully supports Nginx log statistics.

3. Configure Awstats to run automatically
To make the entire log statistical process complete automatically, we need to set the crontab scheduled task to enable Nginx log cutting and Awstats to run automatically and regularly generate the result page.

Vi/etc/crontab
 
11 59 ***/opt/nginx/sbin/logcron. sh # log cutting at pm
 
00 1 ***/usr/local/awstats/tools/awstats_buildstaticpages.pl
-Update-config = www.key0.cn-lang = cn-dir =/data/admin_web/awstats
-Awstatsprog =/usr/local/awstats/wwwroot/cgi-bin/awstats. pl
 
# Log analysis at AM
 
: Wq save and exit
# Crontab/etc/crontab specify the path of the configuration file executed by cron 4. Protect the log statistics result page
Generally, webmasters do not want to let people know the actual traffic of their stations, so they need to use the password protection on the Awstats statistical result page. Nginx uses the same password encryption format as Apache. the built-in apache tool ht is required here.Passwd.
If Apache is installed on the local machine by default, you only need to run it in its program directory.
Example:

/Usr/local/apache2/bin/htpasswd-c admin. pass admin # the user name is admin.
 
New password: enter the password
Re-type new password: Repeated input
Adding password for user admin created successfully
 
Then, hide the password package admin. pass.
Modify nginx. conf to add (BOLD) to location ):
 
Server {
Listen 80;