Goaccess Log Analysis tool using documents

----Sevck 2016/3/4 17:24:13#1软件说明:

Goaccess is an open source, real-time, Web log analysis tool that runs under the command line terminal. The tool provides fast and varied HTTP status statistics, which allows administrators to no longer dwell on the statistics of various types of data, as well as complex instructions and a large stack of piping/regular expressions to say byebye. According to Goaccess official statement: Using AMD Sempron 2.31GHz of CPU+2GB memory, to turn on goaccess all functions, the software can process 100,000 lines per second log. Of course, if you use a more powerful CPU and have more memory, Goaccess will perform better.

#2 Features:

ü Generate statistical data, bandwidth statistics, etc.

ü Time statistics for each request

ü Highest Access IP

ü Requested file statistics

U request static files, pictures, Flash, JS files, etc.

ü Each HTTP status code statistics

ühosts, Reverse Dns,ip location

ü Operating system

ü Browser/Spider

ü References to the website

ü Quoted URLs

ü Keyword Group

U location (Mainland, country, city)

u can output JSON or CSV

ü A variety of color themes

ü Supports high-volume data and supports continuous analysis of large-capacity data

ü Support IPV6

u can generate HTML reports

#3Installation Method:

1. Download the website, compile:

wget http://tar.goaccess.io/goaccess-0.8.1.tar.gz
TAR-XZVF goaccess-0.8.1.tar.gz
CD GOACCESS-0.8.1/
./configure--enable-geoip--enable-utf8
Make && make install

2. Package Manager Installation:

Debian/ubuntu:

#apt-get Install Goaccess

Centos/fedora:

Yum Install goaccess

Os x:

Brew Install Goaccess

Once the installation is complete, we can start using it,

Test whether the installation is successful we can enter in the terminal: goaccess-v (uppercase)

You can see that the installation is successful.

#4启动参数介绍:
-B Traffic Consumption statistics, if you want to increase the resolution speed, do not turn on this option. The default value is not to turn on this option
-F log file path.
-S HTTP Status code report, in order to better parse the log, the option is closed by default.
-e Specifies the access statistics for the IP address.
-A through proxy request statistics.

Menu Introduction

F1 Help Menu
F5 Refresh
Q Exits the current port window, menu, or currently viewed options
o Open the current options, menu
C Change the window color (currently there are only two default and green)
SHIFT + TAB to switch back from the currently selected module
Right open the currently selected module to see more information
S is sorted by date and only works on Access request module
S is sorted by the number of clicks and only works on the Access request module
/view details of the window to search
N finds the location of the next matching content after/, and if not, displays "search hit BOTTOM" at the bottom of the window
T in the View Details window, move the pointer to the top of the
b in the View Details window, move the pointer to the bottom

Other settings

H View Help

V View version

S Storage method

#5使用goaccess分析方法:

-f Specifies that the log file will start the following interface:

SPACEBAR for selection, up and down cursor movement, enter OK

The Nginx log is of type combined log Format (xlf/elf), so we select a third

If you are not nginx, then to vim/etc/goaccess.conf

Found it:

#NCSA Combined Log Format with Virtual Host

#log-format%^:%^%h%^[%d:%^] "%r"%s%b "%r" "%u"

Get rid of the # before Log-format, if not, write a line like the code

Remove the # and change the format.

After the tool is loaded, the page looks like this:

#6键盘操作:

• F1 or H: Help
• F5: Refresh the main interface
• Q: Exit program/Current window/collapse Current module
• O or enter: Expands the selected module or window
• 0-9 and SHIFT + 0: Activates the selected module or window
• K and J: Moving inside the module
• C: Modify Color
• ^f and ^b: scrolling the screen up and down in the module
• TAB SHIFT+TAB: Front and rear switch module
• S: module Internal sorting selection
• /: Search in all modules (support regular)
• N: Find the next match
• G and G: Skip to First/last item

#7功能项:

• Represents a special daily visit
• Request the highest top list
• Static Request Top List
• 404 URL Not Found
• Access the highest IP list
• Operating system
• Browser
• Time statistics
• SOURCE website
• Visit country
• HTTP response Code

#8其他分析技巧:

If your log is a compressed log, you can use the following command:

Zcat access.log.***.gz | Goaccess

# or:

Zcat-f access.log.*.gz | Goaccess

Of course goaccess also supports a variety of pipelines:

For example:

Sed-n '/11/\dec\2016, $p ' Access.log | Goaccess-a

Want to analyze the crawler of Baidu /google website:

Grep-e "(Google|baidu)" Access.log | Goaccess-a

Goaccess also supports generating reports in the Html,csv,json format:

Goaccess-f access.log-a >result.html

Goaccess-f access.log-a-d-o JSON > Result.json

Goaccess-f access.log-o csv > Result.csv

Ps: If you export an HTML report you will encounter the following conditions:

Goaccess-f time_access.log-a > report.html

Goaccess-version 0.9.2-jul 15 2015 16:23:20
Config File:/usr/local/etc/goaccess.conf

Fatal error has occurred
Error occured at:src/parser.c-verify_formats-1691
No time format is found on your conf file.

Workaround:

VI ~/.goaccessrc

Time-format%T
Date-format%d/%b/%y
Log-format%h%^[%d:%t%^] "%r"%s%b "%r" "%u"

#重新指定配置文件后执行
Goaccess-f time_access.log-p ~/.goaccessrc-a > result.html

Finally, attach an HTML report:

Goaccess Log Analysis tool using documents