Automatic analysis of Nginx logs daily with goaccess

Last used awstats from the dynamic analysis Nginx log, but found that it is only cumulative can not see a day or a period of time, so added a goaccess to analyze the log,

Idea: Every day the original Awstats centralized log file processing, generate a new log file in line with the goaccess, and then use goaccess to produce static pages, Nginx will display these static pages.

First, the software

Goaccess:

Ncurses (dependence of goaccess)

Ubuntu:apt-get Install Libncursesw5-dev

Second, installation

wget http://tar.goaccess.io/goaccess-1.1.1.tar.gz tar-zxvf goaccess-1.1.1.tar.gz cd goaccess-1.1.1./configure- Enable-geoip-enable-utf8 make make Install

Third, configuration goaccess.conf

The default goaccess of the Log-format is relatively simple, I have this custom

Time-format%h:%s:%m date-format%d/%b/%y log-format%d:%t%^%H%s%t%M%u%H%b%R%u

The official code for each section of the log, can refer to the revision

Specifiers%x a date and time field matching the time-format and  date-format variables. This is used when a timestamp is  Given instead of the date and time being in two separate  variables.%ttime field matching the time-format variable.%ddate field  matching the date-format variable.%vthe server name according to  the canonical name setting  (server blocks or virtual host).% Ethis is the userid of the person requesting the document as  determined by HTTP authentication.%hhost  (The client ip address,  either ipv4 or ipv6)%rthe request line from the client.  This requires specific delimiters around the request  (As single quotes, double quotes, or  anything else)  to be parsable. if not, we have to use  a combination of special format specifiers as %m %u %h.% Qthe query string.%mthe request method.%uthe url path requested. Note: if the query string is in %u, there is no need  to use %q. however, if the url path, does not include  any query string, you may use %q and the query string  will be appended to the request.%HThe request protocol.%sThe  status code that the server sends back to the client.%bthe  Size of the object returned to the client.%rthe  "Referer"  HTTP request header.%uThe  user-agent http request header.%d The time taken to serve the request, in microseconds.%tthe time  taken to serve the request, in seconds with milliseconds  resolution.%l the time taken to serve the request, in  Milliseconds as a decimal number.%^i Gnore this field.%~move forward through the log string until a  non-space  (!isspace)  char is found.| Vertical pipe or bar is used for either the character prior  to the vertical pipe or followed by the vertical pipe.

• 
  

Iv. Writing scripts

I have this nginx log format is custom, all done the processing. Access logs for different lines are also combined to deal with them.

#!/bin/bashexport path= "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" DATE= '/bin/date-d "1 days ago "+%y%m%d ' prefixs= ' www p m news ' doms= ' xxx.com ' basic_dir="/data/awstats "for PREFIX in $PREFIXSdocat $BASIC _dir/logs/$ Prefix/11/access_$date.log $BASIC _dir/logs/$PREFIX/13/access_$date.log > $BASIC _dir/goaccess/orig_logs/$PREFIX . $DOMS/access_$date.logawk-f "|" ' {print $2,$3,$5,$6,$12,$14,$11,$13} ' $BASIC _dir/goaccess/orig_logs/$PREFIX. $DOMS/access_$date.log > $BASIC _dir /goaccess/logs/$PREFIX. $DOMS/access_$date.log/usr/local/bin/goaccess-f $BASIC _dir/goaccess/logs/$PREFIX. $DOMS/ Access_$date.log-p $BASIC _dir/goaccess/scripts/goaccess.conf > $BASIC _dir/goaccess/html/$PREFIX. $DOMS/access_$ Date.htmldone

Five, crontab automatic processing

Processing starts daily 6 o'clock in the morning

0 6 * * * (bash/data/awstats/goaccess/scripts/goaccess.sh)

VI, NGINX display goaccess results

Cat goaccess.conf

server{Listen 88;        server_name 11.1.2.252;        root/data/awstats/goaccess/html/;        AutoIndex on;        Access_log Logs/access.ga.log; }

Vi. examples

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M01/8A/FE/wKioL1hBIbTD6LH5AATBGMRzmdY420.jpg-wh_500x0-wm_3 -wmp_4-s_117061656.jpg "title=" goaccess.jpg "alt=" Wkiol1hbibtd6lh5aatbgmrzmdy420.jpg-wh_50 "/>

This article is from the "Operation Utopia" blog, please make sure to keep this source http://joeyang.blog.51cto.com/9092193/1878880

Automatic analysis of Nginx logs daily with goaccess