Use GoAccess to analyze Nginx logs and sed/awk manual analysis practices, goaccessng.pdf

Use GoAccess to analyze Nginx logs and sed/awk manual analysis practices, goaccessng.pdf

Preface

Websites using Nginx may encounter access traffic exceptions, friendship detection, program bugs, and other sudden situations. At this time, everyone's response must be the first time to analyze logs, then I found that there are dozens of gigabytes of logs, and whether the information needs to be searched by time, error type, or keyword segment has the illusion of being too tight and the chrysanthemum is tight. The methods described in this article, whether it is GoAccess or sed/awk, can solve temporary problems but may not be able to cure the problem.ELK(Logstash+ElasticSearch+Kibana)Most of us are more rational centralized log management solutions.

Logs are important, but efforts to build a centralized log management platform suitable for business development are the basic core.

Update history

April July 16, 2015-first draft

Http://wsgzao.github.io/post/goaccess/

Additional reading

GoAccess-http://goaccess.io/
Analysis of Nginx log-http://www.fancycoding.com/log-analyse-using-goaccess/ with GoAccess
Sed concise tutorial-http://coolshell.cn/articles/9104.html
AWK concise tutorial-http://coolshell.cn/articles/9070.html

Install GoAccess

Each platform has a very simple deployment scheme-http://goaccess.io/download

wget http://tar.goaccess.io/goaccess-0.9.2.tar.gztar -xzvf goaccess-0.9.2.tar.gzcd goaccess-0.9.2/./configure --enable-utf8makemake install

Usage

For more FAQs, see the official FAQ-http://goaccess.io/faq

# Directly open goaccess-f access. log # select the Log Format NCSA Combined log Format # the remaining operations are quite simple, refer to the extended reading and official documentation # problems with exporting HTML reports goaccess-f time_access.log-a> report.html GoAccess-version 0.9.2-Jul 15 2015 16: 23: 20 Config file: /usr/local/etc/goaccess. confFatal error has occurredError occured at: src/parser. c-verify_formats-1691No time format was found on your conf file. # Add a configuration file vi ~ /. Goaccessrctime-format % Tdate-format % d/% B/% Ylog-format % h % ^ [% d: % t % ^] "% r" % s % B "% R" "% u" # re-specify the configuration file and run goaccess-f time_access.log-p ~ /. Goaccessrc-a> report.html

Use bash/sed/awk to manually find Nginx logs

For more tips, refer to extended reading. The processing efficiency of Python is better.

# Query time periods by date: sed-n "/14 \/Jul \/2015: 00: 00: 00/,/15 \/Jul \/2015: 15: 00: 00/"p access. log> time_access.log # Find the 504 error page and number of awk' ($9 ~ /504 /) 'time_access.log | awk' {print $7} '| sort | uniq-c | sort-rn> 504.log# find the 20 most accessed IP addresses and the number of visits awk' {print $1} 'time_access.log | sort | uniq-c | sort-n-k 1-r | head-n 20> top. log

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.