Rotten mud: awstats is used to analyze nginx logs and awstatsnginx

Rotten mud: awstats is used to analyze nginx logs and awstatsnginx

This document consistsIlanniwebProviding friendship sponsorship, first launchedThe world

For more articles, follow my ilanniweb

Yesterday, nginx logs were cut. For how to cut nginx logs, you can view the article "Bad mud: Cut nginx logs.

We plan to analyze nginx logs today. To analyze nginx logs, we can use shell scripts and third-party software awstats for analysis. Here we choose third-party software awstats for analysis.

To use awstats to analyze nginx logs, We need to install awstats. Before installing awstats, we need to first introduce what awstats is?

I. What is awstats?

Awstats is a free, concise, and powerful Perl-based WEB log analysis tool.

It can collect the following information about the website:

1): visits, visits, page views, clicks, data traffic, etc.

2): accurate to monthly, daily, and hourly data.

3): the visitor's country and IP address.

4): Robots/Spiders statistics.

5): Visitor duration.

6): statistical information of different files types.

7): Pages-URL statistics.

8): Visitor Operating System Browser and other information.

9): Other information (search keyword, etc)

2. Preparations

After introducing what awstats is, let's introduce the preparations before installing awstats.

2.1Install GeoIP

GeoIP is mainly used to locate the access country and city database. You can run the following command to install it:

Yum-y install GeoIP-devel perl-Geo-IP

View the components installed by GeoIP as follows:

Rpm-ql GeoIP-GeoLite-data

Rpm-ql GeoIP-GeoLite-data-extra

For ubuntu, run the following command:

Sudo apt-get-y install geoip-bin geoip-database-contrib php5-geoip python-geoip-dbg libgeoip1 libgeoip-dev

In the above installation file, we only need to pay attention to GeoIP. dat and GeoLiteCity. dat files, GeoIP. dat is used to locate the country to which the visitor belongs, while GeoLiteCity. dat is used to locate the city to which the visitor belongs.

2.2Install httpd-tools

Generally, webmasters do not want others to know the actual traffic of their websites. Therefore, we need to use the awstats statistical result page for password protection.

Nginx uses the same password encryption format as apache, so we need to use htpasswd, which is included in the httpd-tools Component, therefore, we need to install httpd-tools.

Run the following command to install httpd-tools:

Yum-y install httpd-tools

Httpd is automatically installed when httpd-tools is installed, so we need to disable httpd. As follows:

/Etc/init. d/httpd stop

Chkconfig httpd off

To generate a password file, run the following command:

Htpasswd-c-m/usr/local/htpasswd. pass ilanni

3. Install awstats

After the preparation is complete, we will install awstats now.

3.1Download awstats

Awstst official website is http://www.awstats.org/, download the latest version de awstats. As follows:

Wget http://www.awstats.org/files/awstats-7.5.tar.gz

3.2Install awstats

After awstats is downloaded, We will install awstats now. Decompress the installation package and perform the following operations:

Tar-C/usr/local/-xf awstats-7.5.tar.gz

Mv/usr/local/awstats-7.5 // usr/local/awstats

Chown root: root-R/usr/local/awstats/

Mkdir-p/var/www/awstats/

Cd/usr/local/awstats/tools/

Chmod + x/usr/local/awstats/tools/*. pl

Chmod + x/usr/local/awstats/wwwroot/cgi-bin/*. pl

Start to install awstats as follows:

./Awstats_configure.pl

Because nginx is used here, enter none above.

Create a new statistical configuration file.

Enter your own website domain name here, or you can customize it.

Use the default configuration to generate a configuration file.

After the installation program is completed, the corresponding configuration file is generated in the/etc/awstats/directory. As follows:

Ll/etc/awstats/awstats.www.ilanni.com. conf

3.3Edit configuration file

After awstats is installed, We can modify its configuration file awstats.www.ilanni.com. conf. In this configuration file, we mainly modify the path of nginx log analysis by awstats and the log format. Of course, there are some other plug-ins. As follows:

Vim/etc/awstats/awstats.www.ilanni.com. conf

LogFile = "/var/log/nginx/ilanni.com _ % YYYY-24 % MM-24 % DD-24.log"

# Display country and region in configuration

LoadPlugin = "geoip GEOIP_STANDARD/usr/share/GeoIP. dat"

LoadPlugin = "geoip_city_maxmind GEOIP_STANDARD/usr/share/GeoIP/GeoLiteCity. dat"

# Solve garbled search keywords

LoadPlugin = "decodeutfkeys"

Note: The Date Format "% YYYY-24 % MM-24 % DD-24" in the log file in the LogFile path refers to the year, month, and day before 24 hours, that is, the date of yesterday. -24 indicates the day before, and-0 indicates the current.

3.4Install pure IP database plug-in

If you want to display the specific source information of domestic visitors in detail, you can use the pure IP database plug-in.

Pure IP Address Database

After downloading and installing it on a windows machine, the qqwry. dat file under the installation directory is the latest IP address database.

Download the awstats pure IP database parsing plug-in, as shown below:

Http://down.51cto.com/data/1888530

Note that the file paths in the attachment qqhostinfo. pm and qqwry. pl have been modified.

After the download is complete, set qqhostinfo. pm, qqwry. pl, qqwry. dat files are stored in the wwwroot/cgi-bin/plugins directory of awstats and qqhostinfo is given. the executable permission of the pm file. As follows:

Finally, modify the awstats configuration file awstats.www.ilanni.com. conf to add the LoadPlugin = "qqhostinfo" option, as shown below:

The awstats installation and configuration have been completed.

4. Configure the nginx log format

After awstats is installed, We need to configure the nginx log format, because awstats analyzes the nginx log files according to the specified format, so we need to configure the corresponding.

The nginx log configuration method is as follows:

Vim/etc/nginx. conf

Log_format site '$ remote_addr-$ remote_user [$ time_local] "$ request "'

'$ Status $ body_bytes_sent "$ http_referer "'

'"$ Http_user_agent" $ http_x_forwarded_for ';

5. web Integration Between awstats and nginx

After configuring the Nginx log format, we need to integrate awstats with nginx.

Why is awstats integrated with nginx? Because awstats access is performed through the web. This integration can also be divided into two situations: awstats exists separately in the form of nginx virtual host, and awstats is integrated with the existing nginx virtual host. The nginx configurations of the two methods are pasted as follows:

Awstats is used as a virtual host of nginx separately. The configuration file is as follows:

Server {

Listen 80;

Server_name www.ilanni.com;

Root/var/www/awstats /;

Index index.html;

# Or

# Index awstats.test.haiyn.com.html;

Location ~ ^/Cgi-bin/. *. (cgi | pl | py | rb ){

Gzip off;

Include fastcgi_params;

Fastcgi_pass unix:/tmp/php-cgi.sock;

# Fastcgi_pass 127.0.0.1: 9000; # configure fastcgi_pass as your php-fpm server.

Fastcgi_index cgi-bin.php;

Fastcgi_param SCRIPT_FILENAME/usr/local/awstats/wwwroot/cgi-bin/fcgi. php; # Pay Attention to the file path

Fastcgi_param SCRIPT_NAME/cgi-bin/fcgi. php;

Fastcgi_param X_SCRIPT_FILENAME/usr/local/awstats/wwwroot $ fastcgi_script_name; # Pay Attention to the file path

Fastcgi_param X_SCRIPT_NAME $ fastcgi_script_name;

Fastcgi_param REMOTE_USER $ remote_user;

}

# Static awstats files: HTML files stored in DOCUMENT_ROOT/awstats/

Location/classes /{

Alias/usr/local/awstats/wwwroot/classes /;

}

Location/css /{

Alias/usr/local/awstats/wwwroot/css /;

}

Location/js /{

Alias/usr/local/awstats/wwwroot/js /;

}

Location/icon /{

Alias/usr/local/awstats/wwwroot/icon /;

}

}

Integration of awstats with existing nginx Virtual Hosts.

Note: Make sure to store awstats-related configurations to the beginning, because nginx matching rules are matched from top to bottom.

Server {

Listen 80;

Server_name www.ilanni.com ilanni.com;

Charset UTF-8;

Access_log/var/log/nginx/ilanni.com. log main;

Location ~ ^/Icon/{# icon directory

Root/usr/local/awstats/wwwroot;

Index index.html;

Access_log off;

Charset gb2312;

}

Location ~ ^/Cgi-bin/. *. (cgi | pl | py | rb ){

Root/usr/local/awstats/wwwroot;

Gzip off;

Include fastcgi_params;

Fastcgi_pass 127.0.0.1: 9000;

Fastcgi_param X_SCRIPT_NAME $ fastcgi_script_name;

Fastcgi_param SCRIPT_FILENAME $ document_root/cgi-bin/fcgi. php;

Fastcgi_param X_SCRIPT_FILENAME $ document_root $ fastcgi_script_name;

# Fastcgi_param REMOTE_USER $ remote_user;

Auth_basic "Restricted ";

Auth_basic_user_file/usr/local/awstats/htpasswd. pass;

}

Location /{

Root/www/ilanni.com;

Index. php index.html index.htm;

}

Location ~ \. Php $ {

Root/www/ilanni.com /;

Fastcgi_pass 127.0.0.1: 9000;

Fastcgi_index index. php;

Fastcgi_param SCRIPT_FILENAME $ document_root $ fastcgi_script_name;

Include fastcgi_params;

}

}

6. generate an awstats Database

Start to use awstats to analyze logs. Run the following command:

/Usr/local/awstats/wwwroot/cgi-bin/awstats. pl-update-config = www.ilanni.com

To generate a static file, run the following command:

/Usr/local/awstats/tools/awstats_buildstaticpages.pl-update-config = www.ilanni.com-lang = cn-dir =/var/www/awstats-awstatsprog =/usr/local/awstats/wwwroot /cgi-bin/awstats. pl

To view the updated data every day, we can use crontab to regularly update the static page at every day. As follows:

Vim/etc/crontab

1 0 ***/usr/local/awstats/tools/awstats_buildstaticpages.pl-update-config = www.ilanni.com-lang = cn-dir =/var/www/awstats-awstatsprog =/usr/ local/awstats/wwwroot/cgi-bin/awstats. pl

7. Access awstats

After the awstats database is generated, we can access it on the web page. Here I am using the second method to integrate awstats with nginx.

As follows:

Http://www.ilanni.com/cgi-bin/awstats.pl

We entered the username and password generated by htpasswd in Chapter 2 preparation.

The information obtained after awstats analyzes the nginx log file.

It is the country, city, and source address of user access displayed in our installed GeoIP and pure IP database.

Awstats analyzes the keywords searched by the user based on the nginx log file.