Use goaccess to analyze Nginx logs and Sed/awk manual analysis practices

Objective

Use Nginx website may encounter abnormal access traffic, be friendship detection, program appeared bug and other sudden situation, then everyone's reaction must be the first time to analyze the log, and then found that the log has more than dozens of GB, but also need to follow the time, error type or key fields to retrieve information will be a kind of clairvoyant, The illusion of a tight chrysanthemum. The method described in this article, whether it is goaccess or Sed/awk, can solve a temporary problem but may not be able to cure the problems, perhaps ELK(Logstash+ElasticSearch+Kibana) for most of us is more reasonable centralized log management solution.

Log is important, but to build a centralized log management platform suitable for business development is the base core

Update history

July 16, 2015-First draft

Read the original-http://wsgzao.github.io/post/goaccess/

Extended Reading

goaccess-http://goaccess.io/
Using goaccess to analyze Nginx log-http://www.fancycoding.com/log-analyse-using-goaccess/
A concise tutorial on sed-http://coolshell.cn/articles/9104.html
AWK Concise Tutorial-http://coolshell.cn/articles/9070.html

Installing goaccess

Each platform has a grey and simple deployment solution-Http://goaccess.io/download

wget http://tar.goaccess.io/goaccess-0.9.2.tar.gztar -xzvf goaccess-0.9.2.tar.gzcd goaccess-0.9.2/./configure --enable-utf8makemake install

How to use

For more frequently asked questions, please refer to official Faq-http://goaccess.io/faq

#直接打开Goaccess- FAccess.log#选择日志格式NCSA Combined Log Format#剩下的操作都蛮简单的, refer to the extended reading and official documentation.#导出HTML报告会遇到的问题Goaccess- FTime_access.log -A> report.htmlgoaccess-version0.9.2-Jul the  -  -: at: -Config file:/usr/local/etc/goaccess.conffatal error has occurrederror occured at:src/parser.c-verify_formats-1691No time format is found on your conf file.#添加配置文件VI ~/.goaccessrctime-format%tdate-format%d/%b/%ylog-format%h%^[%d:%t%^]"%r"%s%b"%R" "%u"#重新指定配置文件后执行Goaccess- FTime_access.log-p ~/.GOACCESSRC -A> report.html

Using Bash/sed/awk to manually find Nginx logs

More techniques can be referenced by extended reading, Python's processing efficiency or better

#按日期查找时间段"/14\/Jul\/2015:00:00:00/,/15\/Jul\/2015:15:00:00/"p access.log > time_access.log#查找504错误的页面和数量‘($9 ~ /504/)‘‘{print $7}‘504.log#查找访问最多的20个IP及访问次数‘{print $1}‘120 > top.log

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Use goaccess to analyze Nginx logs and Sed/awk manual analysis practices