Nginx Parse Access log file

Nginx Access Log Log Statistics Analysis Common command IP related statistics

Statistics IP Traffic

‘{print $1}‘ access.log | sort -n | uniq | wc -l

View IP traffic for a time period (4-5 points)

grep "07/Apr/2017:0[4-5]" access.log | awk ‘{print $1}‘ | sort | uniq -c| sort -nr | wc -l 

View the top 100 most frequently visited IPs

awk ‘{print $1}‘ access.log | sort -n |uniq -c | sort -rn | head -n 100

View IP access more than 100 times

awk ‘{print $1}‘ access.log | sort -n |uniq -c |awk ‘{if($1 >100) print $0}‘|sort -rn

Query the detailed access of an IP, sorted by frequency of access

grep ‘104.217.108.66‘ access.log |awk ‘{print $7}‘|sort |uniq -c |sort -rn |head -n 100 

Page Access Statistics

View the most frequently accessed pages (TOP100)

awk ‘{print $7}‘ access.log | sort |uniq -c | sort -rn | head -n 100

View the most frequently accessed pages ([Exclude PHP pages "(TOP100)

grep -v ".php"  access.log | awk ‘{print $7}‘ | sort |uniq -c | sort -rn | head -n 100 

View pages that have visited more than 100 times

cat access.log | cut -d ‘ ‘ -f 7 | sort |uniq -c | awk ‘{if ($1 > 100) print $0}‘ | less

View the most recent 1000 records with the highest volume of visited pages

tail -1000 access.log |awk ‘{print $7}‘|sort|uniq -c|sort -nr|less

Request Volume statistics per second

Count requests per second, Top100 Point in time (accurate to seconds)

awk ‘{print $4}‘ access.log |cut -c 14-21|sort|uniq -c|sort -nr|head -n 100

Request Volume statistics per minute

Count requests per minute, Top100 point in time (accurate to minutes)

awk ‘{print $4}‘ access.log |cut -c 14-18|sort|uniq -c|sort -nr|head -n 100

Request volume statistics per hour

Count the number of requests per hour, Top100 point in time (accurate to hours)

awk ‘{print $4}‘ access.log |cut -c 14-15|sort|uniq -c|sort -nr|head -n 100

Performance analysis

Add $request_time to the last field in Nginx log

List pages with transmission time exceeding 3 seconds, showing top 20

cat access.log|awk ‘($NF > 3){print $7}‘|sort -n|uniq -c|sort -nr|head -20

List pages with PHP pages requesting more than 3 seconds, and count the number of times they appear, showing the top 100

cat access.log|awk ‘($NF > 1 &&  $7~/\.php/){print $7}‘|sort -n|uniq -c|sort -nr|head -100

Spider Crawl Stats

Count Spider crawl Times

grep ‘Baiduspider‘ access.log |wc -l

Count spiders to crawl 404 times

grep ‘Baiduspider‘ access.log |grep ‘404‘ | wc -l

TCP Connection Statistics

To view the current number of TCP connections

grep "ESTABLISHED" | grep ":80" | wc -l

Sniff 80-port access with tcpdump to see who is the tallest

80 -c 1000 | awk -F"." ‘{print $1"."$2"."$3"."$4}‘ | sort | uniq -c | sort -nr

Nginx Parse Access log file