Nginx Log Analysis tool goaccess (Turn)

The interview will be quilt cover to the problem is: to give the Web server access logs, please write a script to statistics access to the top 10 of the IP? What are the top 10 requests for access? When you have a taste of goaccess, you understand that these problems, in addition to testing your script memorization ability, the only role is to install a or C.

For nginx log analysis, there are many tools to measure the quality of the standard is probably three fast: fast installation, quick analysis, quick to get started. Satisfying the three-point goaccess is indeed a must-have at home.

Say this title actually a bit wronged goaccess, it is a log analysis tool, not just for nginx use. You can also use it to analyze Apache,iis logs, even your own Web services, if you need to define a parsing log format. As you see a beautiful woman, you'll like it a few features are:

1 Fast parsing speed

2 Easy to use

3 can generate Html,json,csv

Installing goaccess

If you are CentOS, use Yum:

Yum Install goaccess

If your yum can't find goaccess, install Epel first

wget Http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpmwget http://rpms.famillecollet.com /enterprise/remi-release-6.rpmsudo RPM-UVH remi-release-6*.rpm epel-release-6*.rpm

There are two ways to use goaccess, one to display and manipulate directly in the console.

When you want to immediately analyze the IP access situation from 10 to 12 in the morning, knocking commands directly on the console is the quickest.

Another way is to generate HTML files or data files.

If you want to manage the server cluster every day to send you a log of yesterday's access to the situation, this is not the second choice.

Console using Goaccess

Parameters of the Goaccess:

-F <input-file>: Log file that needs parsing-a--agent-list: In the host module is not able to click IP appears this IP user-agentlist, is the function that appears

-C--conf-dialog: whether to display the log and Format Configuration dialog Windows.

This is the window below.

This window has a lot to study.

First, what are the log formats supported by goaccess?

Common Log Format (CLF)

Common log format, example:

127.0.0.1-frank [10/oct/2000:13:55:36-0700] "Get/apache_pb.gif http/1.0" 200 2326 host   user identity author [Date] Request method request  Path request Protocol "Status Code byte number

NCSA commbined Log Format

This is an extension of the common Log format, as an example:

125.125.125.125-dsmith [10/oct/1999:21:15:05 +0500] "get/index.html http/1.0", 1043 "http://www.ibm.com/" mozilla/ 4.05 [en] (WinNT; I) "userid=customera;impid=01234" host user identity author [Date] Request method request path Request Protocol status code byte number referrer client proxy cookie

the

IIS 4.0 and 5.0 are used in this format, as in the following example:

#Software: Microsoft Internet information Server 4.0#version:1.0#date:1998-11-19 22:48:39#fields:date time C-ip Cs-user Name S-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version CS (User-Agent) CS (Cookie) c S (Referrer) 1998-11-19 22:48:39 206.175.82.5-208.201.133.173 get/global/images/navlineboards.gif-200 540 324 157 HTTP/ 1.0 mozilla/4.0+ (compatible;+msie+4.01;+windows+95) userid=customera;+impid=01234/HTTP// Yourturn.rollingstone.com/[email protected] @webx1. html

CloudFront

Log format on AWS

Custom formats

About the parameter description set by custom format here

-D--with-output-resolver Enable IP parsing in HTML output or JSON output

If-d,goaccess is turned on, it will use GEOIP for IP resolution.

-E--exclude-ip=<ip address>

The host module does not require the IP to be counted, such as the ability to filter out the IP of the crawler

-H--help Help Document-H--http-protocol display with HTTP protocol information-M--with-mouse Control Panel on support mouse click, with M, double-click the mouse is equivalent to the operation of the carriage-m--http-method Whether to bring the HTTP method information

The effects of the-H and-m bands are:

-o--output-format=csv|json output to CSV or JSON form-P--conf-file=<filename>

Specify the configuration file, if you have a configuration file, the configuration file also happens to set the Log-format and date, then you will not be forced into the Format dialog box. If you do not have the settings, ~/.GOACESSRC is used by default.

-Q--no-query-string

Ignore parameter part of request

-R--no-term-resolver

Whether IP parsing is performed in the host module. Plus-R is a forbidden IP parsing.

--no-color

Output No color indication

--real-os

Show the real operating system. Whether more detailed operating system information is displayed in the Operation system module.

The following is the keyboard operation of the console:

F1 is to open the Help document

F5 is refreshing the current window

Q is exiting the current window until you exit the program

O is to enter the selection entry

0-9 is the selection module

tab and Shift+tab are toggled before and after the module

J and K are in the module internal entry switch

S is the method of ordering the internal entries of the module

PS: module switching in the upper right corner with the current module indicator

Using goaccess to generate files

The best thing about goaccess is that you can generate HTML, and then you can see a statistical report every day.

The following command can generate an HTML page:

Goaccess-d-F web.log-a-p/home/yejianfeng/.goaccessrc > test.html

The resulting HTML is as follows:

You can also generate JSON:

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o JSON >test.json

and CSV

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o Csv>test.csv

Wait, what's the downside?

Just like the article, no one is perfect, there are no tools, and there are a few places that goaccess.

GeoIP particle size is too coarse

It is the use of the machine comes with the GeoIP, this comes with the IP library of about 77k can only judge the nationality, cannot judge to the city. Well, this means that 99% of the IP stats in your report are from: China. You can upgrade your GeoIP, however, pay. There is a free GeoLiteCity.dat library on the internet, but the goaccess itself does not support nationality-city classification, nor does it support the use of an external IP library.

I can think of another way is to use goaccess to generate JSON, and then write their own python, PHP or program to load the city's IP library to resolve the geographical location. The report is then generated. However, it seems that it is not beautiful.

The good news is that this feature was mentioned in issue, TODO list.

Date granularity is too coarse

Dates can only be granular to days, and if you need to count the data for every hour of the day, you can't. There is no way, you can grep the hours of data, and then use goaccess parsing. However, it seems to be a setback.

The good news is that this feature was mentioned in issue, TODO list.

So.... TODO list is really a good thing, isn't it. In addition, the goaccess of this article uses v0.7.1.

Original address: http://www.cnblogs.com/yjf512/p/3640346.html

Nginx Log Analysis tool goaccess (Turn)