Detailed analysis of Nginx logs using GoAccess

Detailed analysis of Nginx logs using GoAccess

What is GoAccess?

GoAccess is an open-source, real-time web log analysis tool running on the command line terminal. The tool provides fast and diverse HTTP status statistics, so that the Administrator no longer gets stuck with statistics on various types of data, and the complex instructions and a large number of pipelines/regular expressions say byebye.

 

### Function Overview

• Generate Statistics and bandwidth statistics
• Time statistics for each request
• Highest access ip Address
• Request file statistics
• Requested static files, images, flash files, and js files
• HTTP status code statistics
• Hosts, reverse DNS, IP address location
• Operating System
• Browser/spider
• Referenced website
• Referenced URLs
• Keyword Group
• Geographic location (Mainland China, country, city)
• JSON or CSV output
• Various color themes
• Supports large-capacity data and continuous analysis of large-capacity data
• IPv6 support
• Generate HTML reports

### Getting started

• If you want to compile from the source code:

1 2 3 4 5

wget http://tar.goaccess.io/goaccess-0.8.1.tar.gz tar -xzvf goaccess-0.8.1.tar.gz cd goaccess-0.8.1/ ./configure --enable-geoip --enable-utf8 make && make install

• Install from the Package Manager:

Debian/Ubuntu

1	# apt-get install goaccess

CentOS/Fedora

1	yum install goaccess

OSX

1	brew install goaccess

### Use Access to analyze Nginx logs

Then you can start using GoAccess.

First, enter the nginx log directory, such/var/log/nginxRun the following command to enable goaccess to open the specified file:

1	goaccess -f access.log

The page for selecting log file types appears:

Nginx logs belong to the Combined Log Format (XLF/ELF) type, so we select the third one. Move the cursor up and down, select the space, and press enter to confirm.

The main interface is displayed:

Keyboard operation:

• F1Orh: Help
• F5: Refresh the main interface
• q: Exit the program/current window/collapse the current Module
• oOrEnter: Expand the selected module or window
• 0-9AndShift + 0: Activate the selected module or window
• kAndj: Internal module Movement
• c: Modify color
• ^fAnd^b: Scrolling up or down the screen in the module
• tab shift+tab: Front-end switch Module
• s: Module Internal sorting Selection
• /: Search all modules (Regular Expressions supported)
• n: Find the next match
• gAndG: Jump to the first/last item

#### Related to the command line:

Before using the goaccess command line, we must configure the Log File Parsing format, because there is no interface for you to choose the log format.

Take a look at the log format:

1 2	$tail access.log 219.137.254.254 - - [12/Jul/2014:14:55:04 +0800] "GET / HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" "-"

If your log format is the same as mine, you don't have to worry about it yourself. Skip the next section.

If your log format is different from mine, go to the http://goaccess.io/man#custom-log to see the Basic specification of the format. Take my log as an example:

Ip--[date] "request type" Server Response size source browser information "proxy"

%h %^[%d:%^] "%r" %s %b "%R" "%u" %^

The meaning of each representative can be viewed in http://goaccess.io/man#custom-log.

Now configure:

1	vim /etc/goaccess.conf

If this line exists:

1 2	#NCSA Combined Log Format with Virtual Host #log-format %^:%^ %h %^[%d:%^] "%r" %s %b "%R" "%u"

Remove#Right.

If not, write a row.

log-format %h %^[%d:%^] "%r" %s %b "%R" "%u" %^

Find

1 2	#Any Apache log date format #date-format %d/%b/%Y

Cancel the comment of date-format.

Use GoAccess to generate html reports:

1	# goaccess -f access.log -a > report.html

Generate a json report:

1	goaccess -f access.log -a -d -o json > report.json

CSV:

1	goaccess -f access.log -o csv > report.csv

Since nginx will automatically compress logs, the following command can directly analyze the compressed logs:

1 2 3

Zcat access. log. *. gz | goaccess # Or Zcat-f access. log * | goaccess

Supports various pipelines:

1	sed -n '/11\/Dec\/2014/,$ p' access.log | goaccess -a

I want to analyze what Baidu/google has crawled on your website? Easy:

1	grep -E "(google|baidu)"access.log | goaccess -a

Well, Google has already crawled 4 pages on my website today. :)

### What are the shortcomings?

1. Only the geographical database of maxmind is supported. Of course, maxmind also has a free version.

2. I cannot think of other shortcomings. Because pipelines are supported, everything is so convenient.

GoAccess details: click here
GoAccess: click here

This article permanently updates the link address: