Web Access Log Analysis

Use

1. Log the remote host IP address of the access server to see where the visitor came from
2. Record browser access to Web resources to see which parts of the site are most popular
3. Record viewers use the browser to optimize the site based on the browser used by most viewers
4. Record Browser access time

Access Log Location

1.Apache
Look for Customlog "Logs/access.log" in httpd.conf and referenced *.conf files combined
Description

a.CustomLog 访问日志配置指令b.logs/access.log 访问日志记录文件c.combined 日志格式

2. Nginx
   Look in the nginx.conf or referenced *.conf file for Access_log Logs/access.log main
   Description

   a.access_log 访问日志配置指令b.logs/access.log 访问日志记录文件c.main 日志格式

Access log Format

1. Apache

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedLogFormat "%h %l %u %t \"%r\" %>s %b" common

Configuration Instructions

2, Nginx

log_format  main  ‘$remote_addr - $remote_user [$time_local] "$request" ‘              ‘$status $body_bytes_sent "$http_referer" ‘              ‘"$http_user_agent" "$http_x_forwarded_for"‘;

Configuration Instructions
Variable description

3. Common Log Format common

127.0.0.1 - - [14/May/2017:12:45:29 +0800] "GET /index.html HTTP/1.1" 200 4286远程主机IP            请求时间         时区  方法    资源      协议     状态码 发送字节

4. Combined Log format combined

127.0.0.1 - - [14/May/2017:12:51:13 +0800] "GET /index.html HTTP/1.1" 200 4286 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"远程主机IP            请求时间         时区  方法    资源      协议     状态码 发送字节    referer字符           浏览器信息

5. Log Status Code

2XX:

200: 请求成功201: 创建成功202: 接受请求204: 无内容

3XX:

301: 永远重定向302: 临时重定向303: 临时重定向(HTTP1.1 同302)307: 临时重定向(HTTP1.1 POST方法)

4XX:

400: 错误请求401: 访问拒绝403: 访问禁止404: 未找到405: 请求方法错误

5XX:

500: 服务器内部错误503: 服务不可用505: 网关超时

Log statistics

1. View Access IP Address

cat access.log|awk ‘{print $1}‘cat access.log|awk ‘{print $1}‘|sort

2. View the number of visits per IP address

cat access.log|awk ‘{print $1}‘|sort|uniq -ccat access.log|awk ‘{print $1}‘|sort|uniq -c|sort -nrcat access.log|awk ‘{print $1}‘|sort|uniq -c|sort -nr|head -10

3, statistics total number of IP access

cat access.log|awk ‘{print $1}‘|sort|uniq -c|wc -l

4, access the log after the specified time

cat access.log|awk ‘$4>"[23/Aug/2014:23:58:00"‘cat access.log|awk ‘($4>"[23/Aug/2014:23:58:00"){print $1}‘cat access.log|awk ‘($4>"[23/Aug/2014:23:58:00"){print $1}‘|sort|uniq -c|sort -nr

5. Access the log for the specified resource

cat access.log|awk ‘$7 ~/.html$/‘cat access.log|awk ‘($7 ~/.html$/){print $1 " " $7 " " $9}‘cat access.log|awk ‘($7 ~/.js$/){print $10 " " $7}‘|sort|uniq -c|sort -nr|head -10cat access.log|awk ‘($10 > 10000 && $7 ~/.js$/){print $10 " " $7}‘|sort|uniq -c|sort -nr|head -10

6. Total flow of statistics

cat access.log|awk ‘{sum+=$10}END{print sum}‘cat access.log|awk ‘($7 ~/.css$/){sum+=$10}END{print sum}‘grep "04/May/2017" access.log|awk ‘($7 ~/.css$/){sum+=$10}END{print sum}‘

7. Status Code Statistics

cat access.log|awk ‘{print $9}‘ |sort|uniq -c|sort -nrcat access.log|awk ‘($9 ~/^400$/)‘ | wc -lcat access.log | awk ‘($4 ~/^\[04\/May\/2017/){print $9}‘|sort|uniq -c|sort -nrcat access.log | awk ‘$9 ~/400/ && $4 ~/^\[04\/May\/2017/‘|wc -lgrep "04/May/2017" access.log | awk ‘{print $9}‘|sort|uniq -c|sort -nr

Author: KK
Starting Link: https://www.jianshu.com/p/01aa4785f915

Web Access Log Analysis