apache, Nignx and other log analysis tools

Yum Install goaccess

If you are prompted to find a package, you need to install the Epel source First.

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

RPM-UVH epel-release-6*.rpm

or directly RPM-UVH http://mirrors.sohu.com/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm .

Then install it using the Yum install Goaccess.

Installing goaccess

If you are centos, use Yum:

1	yum installgoaccess

If your yum can't find goaccess, install Epel first

1 2 3

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm sudorpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

There are two ways to use goaccess, one to display and manipulate directly in the Console.

When you want to immediately analyze the IP access situation from 10 to 12 in the morning, knocking commands directly on the console is the Quickest.

Another way is to generate HTML files or data Files.

If you want to manage the server cluster every day to send you a log of Yesterday's access to the situation, this is not the second choice.

Console using Goaccess

Parameters of the Goaccess:

-f: log file to parse

-a--agent-list: in the host module is not able to click IP appears this IP user-agentlist, is the function that Appears.

650) this.width=650; "alt=" Image "src=" http://filesimg.111cn.net/2014/08/05/20140805141426737.png "style=" border : 1px solid RGB (230,230,230); margin:0px auto;height:auto; "/>

-c--conf-dialog: whether to display the log and Format Configuration dialog windows.

This is the window below.

650) this.width=650; "title=" image (1) "border=" 0 "alt=" image (1) "width=" 543 "height=" 334 "src="/http Filesimg.111cn.net/2014/08/05/20140805141429855.png "style=" border:1px solid RGB (230,230,230); margin:0px auto; height:auto; "/>

This window has a lot to Study.

first, What are the log formats supported by goaccess?

Common Log Format (CLF)

Common Log format, Example:

127.0.0.1-frank [10/oct/2000:13:55:36-0700] "get/apache_pb.gif http/1.0" 200 2326

Host user Identity author [date] request method request path Request Protocol status code byte number
NCSA commbined Log Format

This is an extension of the common Log format, as an example:

125.125.125.125-dsmith [10/oct/1999:21:15:05 +0500] "get/index.html http/1.0", 1043 "http://www.ibm.com/" mozilla/ 4.05 [en] (WinNT; I) "userid=customera;impid=01234"

Host user Identity author [date] request method request path Request Protocol status code byte number referrer client proxy cookie
W3c

IIS 4.0 and 5.0 are used in this format, as in the following example:

Copy Code
#Software: Microsoft Internet Information Server 4.0

#Version: 1.0

#Date: 1998-11-19 22:48:39

#Fields: Date Time C-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-vers ION CS (user-agent) CS (Cookie) CS (Referrer)

1998-11-19 22:48:39 206.175.82.5-208.201.133.173 get/global/images/navlineboards.gif-200 540 324 157 HTTP/1.0 Mozilla /4.0+ (compatible;+msie+4.01;+windows+95) userid=customera;+impid=01234 Http://yourturn.rollingstone.com/[email protected] @webx1. HTML
Copy Code
CloudFront

Log format on AWS

Custom formats

About the parameter description set by custom format here

-d--with-output-resolver enable IP parsing in HTML output or JSON output

If-d,goaccess is turned on, it will use GeoIP for IP resolution.

-e--exclude-ip=

The host module does not require the IP to be counted, such as the ability to filter out the IP of the crawler

-h--help Help Documentation

-h--http-protocol display with HTTP protocol information

-m--with-mouse Control Panel open support Mouse click, with m, Double-click the mouse is equivalent to the operation of carriage return

-m--http-method whether to bring the HTTP method information

The effects of the-h and-m bands Are:

650) this.width=650; "alt=" Image (2) "src=" http://filesimg.111cn.net/2014/08/05/20140805141431299.png "style=" border:1px solid RGB (230,230,230); margin:0px auto;height:auto;/>

-o--output-format=csv|json output in CSV or JSON format

-p--conf-file=

Specify the configuration file, if you have a configuration file, the configuration file also happens to set the Log-format and date, then you will not be forced into the format dialog box. If you do not have the settings, ~/.goacessrc is used by Default.

-q--no-query-string

Ignore parameter part of request

-r--no-term-resolver

Whether IP parsing is performed in the host Module. Plus-r is a forbidden IP parsing.

--no-color

Output No color indication

--real-os

Show the real operating System. Whether more detailed operating system information is displayed in the operation system Module.

650) this.width=650; "alt=" Image (3) "src=" http://filesimg.111cn.net/2014/08/05/20140805141433238.png "style=" border:1px solid RGB (230,230,230); margin:0px auto;height:auto;/>

The following is the keyboard operation of the Console:

F1 is to open the Help document

F5 is refreshing the current window

Q is exiting the current window until you exit the program

O is to enter the selection entry

0-9 is the selection module

tab and Shift+tab are toggled before and after the module

J and K are in the module internal entry switch

S is the method of ordering the internal entries of the module

Ps: module switching in the upper right corner with the current module indicator

Using goaccess to generate files

The best thing about goaccess is that you can generate html, and then you can see a statistical report every DAY.

650) this.width=650; "title=" image (5) "border=" 0 "alt=" image (5) "width=" 763 "height=" 412 "src="/http Filesimg.111cn.net/2014/08/05/20140805141435996.png "style=" border:1px solid RGB (230,230,230); margin:0px auto; height:auto; "/>

How to Use:

Interactive reports that appear in terminal

# Goaccess-f/data/wwwlogs/access_nginx.log

To generate an HTML report:

# goaccess-f access.log-a > report.html

To generate a JSON report:

# goaccess-f Access.log-a-d-o JSON > Report.json

To generate a CSV report:

# goaccess-f Access.log--no-csv-summary-o csv > Report.csv

This article is from the "dream to reality" blog, Please be sure to keep this source http://lookingdream.blog.51cto.com/5177800/1880740

apache, Nignx and other log analysis tools