[Apache] uses Shell to analyze website access

With the website running, we can get some actual information about the website through the free Log analysis tool such as awstats, such as the daily IP amount, PV amount, the browser used by the user, the operating system used by the user, etc., but sometimes want to get some information from the Web log file by hand. , here are some of the commands I used most recently
Get access to the top 10-bit IP address
Cat Access.log|gawk ' {print $} ' |sort|uniq-c|sort-nr |head-10
Most visited files or pages
Cat Access.log|gawk ' {print $11} ' |sort|uniq-c|sort-nr
By the number of sub-domain access, based on referer to calculate, slightly not allowed
Cat Access.log | awk ' {print $11} ' | Sed-e ' s/http:\/\///'-e ' s/\/.*//' | Sort | uniq-c | Sort-rn | Head-20
List several files with the largest transfer size
Cat Www.access.log |awk ' ($7~/\.php/) {print $ "" $ "" $4 "" $7} ' |sort-nr|head-100
Lists pages with output greater than 200000byte (about 200kb) and the number of occurrences of the corresponding page
Cat Www.access.log |awk ' ($ > 200000 && $7~/\.php/) {print $7} ' |sort-n|uniq-c|sort-nr|head-100
If the last column of the log records the paging file transfer time, there are the most time-consuming pages listed to the client
Cat Www.access.log |awk ' ($7~/\.php/) {print $NF "" $ "" $4 "" $7} ' |sort-nr|head-100
List the most time-consuming pages (more than 60 seconds) and the number of corresponding page occurrences
Cat Www.access.log |awk ' ($NF > && $7~/\.php/) {print $7} ' |sort-n|uniq-c|sort-nr|head-100
List files that have been transmitted for longer than 30 seconds
Cat Www.access.log |awk ' ($NF >) {print $7} ' |sort-n|uniq-c|sort-nr|head-20
Lists the number of runs per process for the current server, in reverse order
Ps-ef | Awk-f ' {print $8 "" $9} ' |sort | Uniq-c |sort-nr |head-20

 1. See how many IP accesses are available on the day:
awk ' {print '} ' log_file|sort|uniq|wc-l
2. View the number of times a page has been accessed:
grep "/index.php" Log_file | Wc-l
3. See how many pages each IP visited:
awk ' {++s[$1]} END {for (a in S) print A,s[a]} ' log_file
4, the number of pages per IP access from small to large sort:
awk ' {++s[$1]} END {for (a in S) print S[a],a} ' log_file | Sort-n
5. See which pages a certain IP has accessed:
grep ^111.111.111.111 log_file| awk ' {print $1,$7} '
6, remove the search engine statistics on the day of the page:
awk ' {print $12,$1} ' log_file | grep ^\ "Mozilla | awk ' {print $} ' |sort | Uniq | Wc-l
7. View June 21, 2009 14 O'Clock How many IP accesses are in this one-hour period:
awk ' {print $4,$1} ' log_file | grep 21/jun/2009:14 | awk ' {print $} ' | Sort | Uniq | Wc-l
8-------------------------------------------------
[Log analysis] Shell statistics log client IP total traffic sequencing
#linux Apache Nginx
awk ' {a[$1]=a[$1]+$10}end{for (i in a) print a/1024/1024 "MB", i} ' Access_bbs.linuxtone.org.20110114|sort-nr >ip_ tarffic_bbs.linuxtone.org.20110114
#iis
awk ' {a[$7]=a[$7]+ $NF}end{for (i in a) print a/1024/1024 "MB", i} ' Access_bbs.linuxtone.org.20110114|sort-nr >ip_ tarffic_bbs.linuxtone.org.20110114
Specific time period:
#linux Apache Nginx
Sed-n '/\[13\/jan\/2011\:16\:00\:00/,/\[13\/jan\/2011\:19\:00\:00/p ' Access_bbs.linuxtone.org.20110114|awk ' {a[$1 ]=a[$1]+$10}end{for (i in a) print a[i]/1024/1024 "MB", i} ' |sort-nr >ip_tarffic_bbs.linuxtone.org.20110114_16-18
#iis
Sed-n '/16\:00\:00/,/19\:00\:00/p ' Access_bbs.linuxtone.org.20110114|awk ' {a[$7]=a[$7]+ $NF}end{for (i in a) print a[i] /1024/1024 "MB", i} ' |sort-nr >ip_tarffic_bbs.linuxtone.org.20110114_16-18

[Apache] uses Shell to analyze website access