The latest log analysis tool Awstats: Nginx

Preface:

Awstats is a fast-growing Perl-based WEB log analysis tool on SourceForge. A full log analysis allows Awstats to show you the following information:

Number of visits, unique visitor count, access time and last visit, user authentication, recent authenticated access, weekly peak hours (pages, click rate, kilobytes per hour and week ), host Guest for domain name/country (page number, click rate, byte, 269 domain name/country detection, geoip Detection), host list, most of the recently accessed and unresolved IP address lists read In and out pages, file types, website compression statistical tables (mod_gzip or mod_deflate), operating systems used (pages of each operating system, click rate, byte, 35 OS detected), browser used, robot access (detect 319 robots), worm attack (five worm families), search engine, search for your address using keywords, HTTP protocol errors (pages not found recently), other URL-based Personalized reports, link parameters, involving integrated marketing purposes. your website is added to "favorite bookmarks ". times. screen size (HTML tags need to be added on the index page ). browser support ratio: Java, Flash, RealG2 reader, Quicktime reader, WMA reader, PDF reader. server Load balancer server ratio cluster report.

The running of Awstats requires support from the PERL environment. From the awstats documentation, it provides excellent support for Apache HTTP Server. When we replace the Web Server with Nginx, it is troublesome to run awstats. Nginx has poor support for Perl, and is not even recommended by the official team. In addition, you need to modify the log format before running Nginx.

You can use awstats to analyze apache logs or nginx logs. This topic describes how to automatically and periodically cut nginx access logs, and uses awstats to periodically analyze nginx logs and implement statistical results for security and convenience.

Environment:

CentOS 6.4x86 _ 64ip: 192.168.1.113 Domain Name: www.sunsky.com (server and client are resolved through the hosts file) nginx-1.2.9 compilation and installation, path/usr/local/nginx, the default log format of the Service Activation status is nginx. Do not change it. Otherwise, awstats cannot analyze logs. Log_format main '$ remote_addr-$ remote_user [$ time_local] "$ request" ''$ status $ body_bytes_sent" $ http_referer "'' "$ http_user_agent" "$ http_x_forwarded_for login"

I. Automatic Log Cutting

For nginx log cutting, since the cronolog tool is not used like apache, We will write a script so that it can be automatically executed at every day, cut logs from yesterday (analyzed by awstats) and compress logs from the day before yesterday (compressed logs can reduce storage space. To prevent awstats from being compressed after analysis, only logs from the day before yesterday are compressed ).

vim /server/scripts/cut_nginx_log.sh

Enter the following content:

#!/bin/shyesterday=`date-d "yesterday"+"%Y%m%d"`before_yesterday=`date-d "-2 day"+"%Y%m%d"`Nginx_Dir="/usr/local/nginx"Nginx_logs="/app/logs"Log_Name="www_access"cd/tmp[ -d $Nginx_Logs ] && cd$Nginx_logs || exit1[ -f $Log_Name.log ] && /bin/mv$Log_Name.log ${Log_Name}_${yesterday}.log || exit1if[ $? -eq0 -a -f $Nginx_Dir/logs/nginx.pid ]thenkill-USR1 `cat$Nginx_Dir/logs/nginx.pid`fi[ -f ${Log_Name}_${before_yesterday}.log ] && /usr/bin/gzip${Log_Name}_${before_yesterday}.log|| exit1

Execute crontab-e to add the script to the scheduled task.

1 0 * * * /bin/sh /server/scripts/cut_nginx_log.sh >/dev/null 2>&1

In this way, log cutting, compression, and other functions can be automatically implemented at every day.

Because nginx in this experiment already has logs, and awstats can analyze the cut logs, we need to run this script here, to cut existing logs and generate the logs generated yesterday to facilitate subsequent operations.

/bin/sh/server/scripts/cut_nginx_log.sh >/dev/null2>&1

Ii. installation and configuration of Awstats

1. Deploy awstats

First, we need to download the awstats package and put it in the regular directory (/usr/local ).

wget http://awstats.sourceforge.net/files/awstats-7.2.tar.gztarzxf awstats-7.2.tar.gzmvawstats-7.2 /usr/local/awstats

Because the permission in the downloaded package of wget is not root, you must modify the permission here. Otherwise, *. pl cannot run later.

chown-R root.root /usr/local/awstatschmod+x /usr/local/awstats/tools/*.plchmod+x /usr/local/awstats/wwwroot/cgi-bin/*.pl

Next, run the awstats_configure.pl Configuration Wizard under awstats/tools to generate the awstats configuration file. The naming rule of the awstats configuration file is awstats. website. conf.

cd/usr/local/awstats/tools/./awstats_configure.pl

The following prompt appears:

----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur ----- This tool will help you to configure AWStats to analyze statistics forone web server. you can try to use it to letit doall that is possibleinAWStats setup, however following the step by step manual setupdocumentation (docs/index.html) is often a better idea. above all if:-You are not an administrator user,-You want to Alyze downloaded log files without web server,-You want to analyze mail or ftplog files instead of web log files,-You need to analyze load balanced servers log files, -You want to 'understamp' all possible ways to use AWStats... read the AWStats documentation (docs/index.html ). -----> Running OS detected: Linux, BSD or Unix -----> Check forweb server installEnter full config filepath of your Web ser Ver. example:/etc/httpd. confExample:/usr/local/apache2/conf/httpd. confExample: c: \ Program files \ apache group \ apache \ conf \ httpd. confConfig filepath ('none' to skip web server setup):> none # Here Let's fill in the configuration file path of the web server, because we are not using apache, therefore, enter noneYour web server config file (s) cocould not be found. you will need to setup your web server manually to declareAWStatsscript as a CGI, ifyou want to build reports dy Namically. see AWStats setup documentation (filedocs/index.html) -----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats. model. conf 'file awstats. model. conf updated. -----> Need to create a new config file? Do you want me to build a new AWStats config/profilefile (required iffirst install) [y/N]? Y # check whether a new configuration file is created. Enter y -----> Define config filename to createWhat is the name of your web site or profile analysis here? Example: www. mysite. comExample: demoYour web site, virtual server or profile name:> www.sunsky.com # Enter your website domain name here, virtual Host name or any configuration name -----> Define config filepathIn whichdirectory doyou plan to store your config file (s )? Default:/etc/awstatsDirectory path to store config file (s) (Enter fordefault): ># Enter the path of your configuration file, we use the default path/etc/awstats, so press enter to -----> Create config file '/etc/awstats/awstats.www.sunsky.com. conf 'config file/etc/awstats/awstats.www.sunsky.com. conf created. -----> Add update process inside a schedulerSorry, configure. pl does not support automatic add to cronyet. you can doit manually by adding the following commandto your cron:/usr/local/awstats/wwwroot/cgi-bin/awstats. pl-update-config = www. sunsky. comOr ifyou have several config files and prefer having only one command:/usr/local/awstats/tools/awstats_updateall.pl nowPress ENTER to continue... # prompt that the crontab scheduled task cannot be automatically added. You need to add it later. Press enter to continue a simple config filehas been created:/etc/awstats/awstats.www.sunsky.com. confYou shoshould have a lookinside to check and change manually main parameters. you can thenmanually update your statistics for 'www .sunsky.com 'with command:> perl awstats. pl-update-config = www. sunsky. comYou can also build static report pages for 'www .sunsky.com 'with command:> perl awstats. pl-output = pagetype-config = www. sunsky. comPress ENTER to finish... # A message is displayed, indicating that the configuration file is created and how to update the configuration and create a static report page. Press enter to complete the Configuration Wizard.

2. Modify the awstats configuration file

After the configuration file is created, we also need to modify some parameters in/etc/awstats/awstats.www.sunsky.com. conf.

sed-i 's#LogFile="/var/log/httpd/mylog.log"#LogFile="/app/logs/www_access_%YYYY-24%MM-24%DD-24.log"#g'/etc/awstats/awstats.www.sunsky.com.conf

The purpose of this change is to specify the path of the nginx log file to be analyzed by awstats. The path here must be set according to your Log Path.

1sed -i 's#DirData="/var/lib/awstats"#DirData="/usr/local/awstats/data"#g'/etc/awstats/awstats.www.sunsky.com.conf

The purpose of this change is to specify the awstats database configuration file (that is, the awstats database (plain text )). Because there is no/usr/local/awstats/data directory, we need to create

1mkdir/usr/local/awstats/data

After the above two replacement operations are completed, you must run the command to check whether the replacement is successful, so as to detect leakage early.

12grep"LogFile="/etc/awstats/awstats.www.sunsky.com.confgrep"DirData="/etc/awstats/awstats.www.sunsky.com.conf

After the query replacement result is correct, perform the following steps.

3. Create a directory for storing awstats statistics

Now we want to create a database directory for storing the awstats statistical results

1/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.sunsky.com

If the screen output is similar to the following prompt, the configuration file is correct.

123456789101112Create/Updatedatabase forconfig "/etc/awstats/awstats.www.sunsky.com.conf"by AWStats version 7.2 (build 1.992)From data inlog file"/app/logs/www_access.log"...Phase 1 : First bypass old records, searching new record...Searching new records from beginning of log file...Jumped lines infile: 0Parsed lines infile: 0Found 0 dropped records,Found 0 comments,Found 0 blank records,Found 0 corrupted records,Found 0 old records,Found 0 new qualified records.

Note: awstats. pl will search in the/etc/awstats directory, read the parameters in-config according to the program running the read configuration file, and extend www.sunsky.com to awstats.www.sunsky.com. conf.

Log Analysis: after running, the log statistics are archived into an awstats database (plain text;

Then there is the output in two forms:

1. One is to read the output of the statistical result database through cgi program;

2. Run the background script to export the output to a static file;

After the statistical analysis is complete, the results are still in the Awstats database. On Apache, you can directly open the Perl program webpage to view statistics. However, Nginx does not support Perl well. Therefore, we need to use the awstats tool to generate static files based on the statistical results. here we can use the script to implement it easily.

1vim /server/scripts/awstats.sh

Enter the following content

12345#！/bin/shAwstats_Dir="/usr/local/awstats"[ -d /www/awstats]||mkdir/www/awstats$Awstats_Dir/wwwroot/cgi-bin/awstats.pl -update -config=www.sunsky.com$Awstats_Dir/tools/awstats_buildstaticpages.pl -update -config=www.sunsky.com -awstatsprog=$Awstats_Dir/wwwroot/cgi-bin/awstats.pl -lang=cn -dir=/www/awstats

Script content:

12345/usr/local/awstats/tools/awstats_buildstaticpages. Pl Awstats static page generation tool-Update-config = www.sunsky.com: update Configuration items-awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats. Pl Awstats log update program path-Lang = cn: Chinese-dir= /www/awstatsStatistics output directory

Awstats_buildstaticpages.pl reads the database configuration file in the DirData path based on the-config parameter, generates a static html file, and redirects the generated file to the/www/awstats directory.

3. Configure nginx for secure access

Next, we need to configure nginx so that it can securely access the analyzed data.

1vim /usr/local/nginx/conf/nginx.conf

Add the following content to server:

123456789101112131415161718192021222324server {listen 80;server_name www.sunsky.com;location / {root /www/sunsky;index index.html index.htm;access_log /app/logs/www_access.log commonlog;}location ~ ^/awstats/{root /www/;index awstats.www.sunsky.com.html; # Modify the homepage file based on your website Domain Nameautoindex on；access_log off;charset gb2312;auth_basic "Restricted"; # Some websites do not want to disclose website traffic information, so add an authenticationauth_basic_user_file /usr/local/nginx/htpasswd.pass; # This file is created by apache's encryption and authentication tool htpasswd.}location ~ ^/icon/{root /usr/local/awststs/wwwroot;index index.html;access_log off;charset gb2312;}}

Because nginx does not have a good encryption and authentication tool, apache's htpasswd must be used for encryption and authentication.

1htpasswd -c -m /usr/local/nginx/htpasswd.pass sunskyadmin # The username is sunskyadmin.

After the configuration is complete, check the nginx syntax, and then use the browser to access www.sunsky.com/awstats. after entering the account number password, you can collect statistics.

So far, awstats has been able to implement Nginx log statistics and static security access functions.

4. Configure awstats to run automatically

To automate the entire log statistical process. THE sh script is added to the crontab scheduled task. In this case, combined with the preceding scheduled cut task, two more scheduled tasks will be added to the crontab.

1210* * * /bin/sh /server/scripts/cut_nginx_log.sh >/dev/null2>&101* * * /bin/sh /server/scripts/awstats.sh >/dev/null2>&1

At this point, the configuration of the Log Access Tool awstats on nginx is complete. Of course, this article is based on the display of static pages. I will write a blog post about the log analysis tool Awstats in practice. The analysis results will be dynamic and shared with you.

This article is from the "sunsky" blog, please be sure to keep this source http://sunsky.blog.51cto.com/4824967/1316979