Log Analysis Tool--goaccess

Send log analysis results to boss and research and development on a daily basis, is it time to write your own scripts?

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/4C/BC/wKioL1REr0SyFyRrAAbfTaYWVPI456.jpg "title=" 1.jpg " alt= "Wkiol1rer0syfyrraabftaywvpi456.jpg"/>

Here is the goaccess installation steps, I also grabbed from other places, installation is very simple, a search on the internet a lot

Installing goaccess

Redhat or CentOS, using yum:

Yum Install goaccess

There are two ways to use goaccess, one to display and manipulate directly in the console.

When you want to immediately analyze the IP access situation from 10 to 12 in the morning, knocking commands directly on the console is the quickest.

Another way is to generate HTML files or data files.

If you want to manage the server cluster every day to send you a log of yesterday's access to the situation, this is not the second choice.

Console using Goaccess

Parameters of the Goaccess:

-F <input-file>: Log file that needs parsing-a--agent-list: In the host module is not able to click IP appears this IP user-agentlist, is the function that appears

650) this.width=650; "title=" Image "src=" Http://images.cnitblog.com/blog/136188/201404/021206455628865.png "alt=" Image "width=" 732 "height=" 235 "border=" 0 "style=" border:0px; "/>

-C--conf-dialog: whether to display the log and Format Configuration dialog Windows.

This is the window below.

650) this.width=650; "Title=" Image (1) "src=" Http://images.cnitblog.com/blog/136188/201404/021206465465166.png "alt = "Image (1)" width= "543" height= "334" border= "0" style= "border:0px;"/>

This window has a lot to study.

First, what are the log formats supported by goaccess?

Common Log Format (CLF)

Common log format, example:

127.0.0.1-frank [10/oct/2000:13:55:36-0700] "Get/apache_pb.gif http/1.0" 200 2326 Host user identity author [date] "Request method request path Request Protocol" form State Code byte number

NCSA commbined Log Format

This is an extension of the common Log format, as an example:

125.125.125.125-dsmith [10/oct/1999:21:15:05 +0500] "get/index.html http/1.0", 1043 "http://www.ibm.com/" mozilla/ 4.05 [en] (WinNT; I) "userid=customera;impid=01234" host user identity author [Date] Request method request path Request Protocol status code byte number referrer client proxy cookie

the

IIS 4.0 and 5.0 are used in this format, as in the following example:

650 "this.width=650;" src= "http:// Common.cnblogs.com/images/copycode.gif "alt=" Copy Code "style=" Border:none rgb (221,221,221); Background-color:rgb ( 255,255,255); "/>

 #Software:  microsoft internet information server 4.0#version: 1.0#date: 1998-11-19  22:48:39#fields: date time c-ip cs-username s-ip cs-method cs-uri-stem  cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version cs ( user-agent)  cs (Cookie)  cs (Referrer) 1998-11-19 22:48:39 206.175.82.5 -  208.201.133.173 get /global/images/navlineboards.gif - 200 540 324 157  http/1.0 mozilla/4.0+ (compatible;+msie+4.01;+windows+95)  USERID=CustomerA;+IMPID=01234  Http://yourturn.rollingstone.com/[email protected] @webx1. html 

650) this.width=650; "src=" Http://common.cnblogs.com/images/copycode.gif "alt=" Copy Code "style=" Border:none RGB ( 221,221,221); Background-color:rgb (255,255,255); "/>

CloudFront

Log format on AWS

Custom formats

About the parameter description set by custom format here

-D--with-output-resolver Enable IP parsing in HTML output or JSON output

If-d,goaccess is turned on, it will use GEOIP for IP resolution.

-E--exclude-ip=<ip address>

The host module does not require the IP to be counted, such as the ability to filter out the IP of the crawler

-H--help Help Document-H--http-protocol display with HTTP protocol information-M--with-mouse Control Panel on support mouse click, with M, double-click the mouse is equivalent to the operation of the carriage-m--http-method Whether to bring the HTTP method information

The effects of the-H and-m bands are:

650) this.width=650; "Title=" Image (2) "src=" Http://images.cnitblog.com/blog/136188/201404/021206478753193.png "alt = "Image (2)" width= "749" height= "209" border= "0" style= "border:0px;"/>

-o--output-format=csv|json output to CSV or JSON form-P--conf-file=<filename>

Specify the configuration file, if you have a configuration file, the configuration file also happens to set the Log-format and date, then you will not be forced into the Format dialog box. If you do not have the settings, ~/.GOACESSRC is used by default.

-Q--no-query-string

Ignore parameter part of request

-R--no-term-resolver

Whether IP parsing is performed in the host module. Plus-R is a forbidden IP parsing.

--no-color

Output No color indication

--real-os

Show the real operating system. Whether more detailed operating system information is displayed in the Operation system module.

650) this.width=650; "Title=" Image (3) "src=" Http://images.cnitblog.com/blog/136188/201404/021206491565235.png "alt = "Image (3)" width= "742" height= "391" border= "0" style= "border:0px;"/>

The following is the keyboard operation of the console:

F1 is to open the Help document

F5 is refreshing the current window

Q is exiting the current window until you exit the program

O is to enter the selection entry

0-9 is the selection module

tab and Shift+tab are toggled before and after the module

J and K are in the module internal entry switch

S is the method of ordering the internal entries of the module

PS: module switching in the upper right corner with the current module indicator

Using goaccess to generate files

The best thing about goaccess is that you can generate HTML, and then you can see a statistical report every day.

The following command can generate an HTML page:

Goaccess-d-F web.log-a-p/home/yejianfeng/.goaccessrc > test.html

The resulting HTML is as follows:

650) this.width=650; "Title=" Image (5) "src=" Http://images.cnitblog.com/blog/136188/201404/021206510622919.png "alt = "Image (5)" width= "763" height= "412" border= "0" style= "border:0px;"/>

You can also generate JSON:

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o JSON >test.json

and CSV

Goaccess-q-F web.log-a-p/home/yejianfeng/.goaccessrc-o Csv>test.csv

I usually save the analysis results in HTML format and then send it through a Python script with the following script:

#!/usr/bin/python#coding:utf-8import smtplibimport sysfrom email. header import headerfrom email.mime.text import mimetextfrom  Email.mime.multipart import mimemultipartfrom email.mime.image import mimeimagedef  sendmail (sender,receiver,subject,attfile):    smtpserver =  ' smtp.126.com '     username =  ' admin123 '     password =  ' 111111 '     msg = mimemultipart (' alternative ')     msg[' Subject ']  = header (Subject, ' Utf-8 ')     f = open (attfile)      html =  ". Join (F.readlines ())     f.close ()     part2  = mimetext (HTML, ' HTML ', ' Utf-8 ')     msg.attach (part2)     att  = mimetext (Open (Attfile, ' RB '). Read (), ' base64 ', ' Utf-8 ')     att["Content-type"] =  ' Application/octet-stream '     att[" Content-disposition "] =  ' attatchment;filename=" report.html "'     msg.attach (ATT)     smtp = smtplib. SMTP ()     smtp.connect (' smtp.126.com ')     smtp.login (Username,password )     smtp.sendmail (sender,receiver,msg.as_string ())     smtp.quit () if  __name__ ==  ' __main__ ':    sender =  ' [email protected] '     receiver = [' [email protected] ', ' [email protected] ']     subject =  ' xxx log Analysis report '     sendmail (sender,receiver,subject,sys.argv[1 ])

This article is from the "Zhaohaihua _ Yun-dimensional Road" blog, please be sure to keep this source http://baiying.blog.51cto.com/1068039/1565880

Log Analysis Tool--goaccess