The use method of Nginx log Analysis tool goaccess in Linux

The interview will be Beimian to the problem is: give the Web server access log, please write a script to statistics access to the top 10 IP? What are the top 10 requests for access? When you get to know the goaccess, you'll understand that the only thing that can be done is to put a or C in addition to the ability to memorize your scripts.

For nginx log analysis, there are many tools, measuring the standard of good or bad is probably three fast: Install fast, parse fast, start fast. Meeting these three points of goaccess is indeed the home of the necessary medicine.

In fact, the title is a bit wronged goaccess, it is a log analysis tool, not just for nginx use. You can also use it to analyze Apache,iis logs, or even Web services you write yourself, assuming that you need to define a good log format for parsing. If you see a beautiful woman, you will look like a few of its functions are:

1 parsing speed is fast

2 Easy to use

3 can generate Html,json,csv

Install Goaccess

If you are CentOS, use Yum:

1	yum install goaccess

If your yum can't find goaccess, install Epel first.

1 2 3

wget http: //rpms . Famillecollet.com /enterprise/remi-release-6 . RPM sudo RPM-UVH remi-release-6*.rpm epel-release-6*.rpm

There are two ways to use goaccess, one that is displayed and manipulated directly on the console.

When you want to immediately analyze the IP access situation at 10 to 12 this morning, it's fastest to hit the console directly.

Another is to generate HTML files or data files.

If you want to manage the server cluster every day to send you a visit to the log yesterday, this is not the second choice.

Console uses goaccess

Parameters for Goaccess:

-F: Log files that need to be resolved

-A--agent-list: In the host module is not able to click IP to appear this IP user-agentlist, is to appear as the function of the diagram.

-C--conf-dialog: whether to display the log and Format Configuration dialog window.

This is the window below.

This window has a lot to study.

First, what are the log formats supported by goaccess?

Common Log Format (CLF)

Common log format, example:

127.0.0.1-frank [10/oct/2000:13:55:36-0700] "Get/apache_pb.gif http/1.0" 200 2326

Host user identity author [date] "Request method request path Request Protocol" status Code byte number
NCSA commbined Log Format

This is an extension of the common Log format, for example:

125.125.125.125-dsmith [10/oct/1999:21:15:05 +0500] "get/index.html http/1.0 1043" http://www.ibm.com/"" mozilla/ 4.05 [en] (WinNT; I) "userid=customera;impid=01234"

Host user identity author [date] "Request method request path Request Protocol" status code byte number referrer client proxy cookie
W3c

IIS 4.0 and 5.0 are used in this format, as in the following example:

Copy Code
#Software: Microsoft Internet Information Server 4.0

#Version: 1.0

#Date: 1998-11-19 22:48:39

#Fields: Date Time C-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-vers Ion CS (user-agent) CS (Cookie) CS (referrer)

1998-11-19 22:48:39 206.175.82.5-208.201.133.173 get/global/images/navlineboards.gif-200 540 324 157 Mozilla /4.0+ (compatible;+msie+4.01;+windows+95) userid=customera;+impid=01234 http://yourturn.rollingstone.com/webx?98@ @webx1. html
Copy Code
CloudFront

Log format on AWS

Custom Format

A description of the parameters set for the custom format here

-d--with-output-resolver Open IP resolution in HTML output or JSON output

If-d,goaccess is turned on, it will use GEOIP for IP resolution.

-E--exclude-ip=

The host module does not need to be a statistical IP, for example, you can filter out the IP of the crawler

-H--help Help document

-H--http-protocol with HTTP protocol information when displayed

-M--with-mouse Control Panel open support mouse click, with the M, double-click the mouse is equivalent to the operation of the carriage return

-M--http-method whether to bring HTTP method information

The effect of the-H and-M band is:

-o--output-format=csv|json output to CSV or JSON form

-P--conf-file=

Specify the configuration file, and if you have a configuration file and you happen to have Log-format and date set in the configuration file, you will not be forced into the Format dialog box. If you don't have it set, use ~/.GOACESSRC by default.

-Q--no-query-string

Ignore the parameter portion of the request

-R--no-term-resolver

Whether IP parsing is performed in the host module. Plus-R prohibits IP resolution.

--no-color

No color indicator for output

--real-os

Show the real operating system. Whether to show more detailed operating system information in the Operation system module.

As shown in figure:

The following is a keyboard operation for the console:

F1 is to open the Help document

F5 is refreshing the current window

Q is to exit the current window until the program is exited

O is to enter the selection entry

0-9 is the selection module

tab and Shift+tab are switched before and after the module

J and K are the entry switches within the module

S is the way to sort the items inside the module

PS: module switch in the upper right corner of the current module is marked

Using goaccess to generate files

Speaking of goaccess the most cool thing is that you can generate HTML, and then every day you can see a statistical report.

The following command generates an HTML page:

1	goaccess -d -f web.log -a -p /home/yejianfeng/ .goaccessrc > test .html

The screenshot of the generated HTML is as follows:

Can also generate JSON:

can also generate JSON:

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o JSON >t Est.json

and CSV

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o csv>test.csv