Common Linux website log analysis commands

1. Enter the log storage folder

Cd d:/111

2. view all documents under this file

Ls

3. Merge logs or other files

Cat *. Log> example. log # merge log files suffixed with log

Cat 1.log 2.log> 3.log # merge specified logs

4. Extract Baidu Spider (if there is an error, please refer to: http://www.lirang.net/post/38.html)

Grep "baiduspider" example. Log> baiduspider.txt # extract 404, 500, and so on.

5. Extract things that match both attributes at the same time

Egrep "baiduspider | googlebot" example. Log> spider.txt # others and so on

6. Extract all status codes and quantities generated by Baidu spider access and sort them in descending order.

Cat *. log | grep 'baidider '| awk' {print $11} '| sort | uniq-c | sort-Nr | awk' {print $2 "\ t" $1 }'> baiduma. log

Or: awk '{print $11}' Baidu. log | sort | uniq-c | sort-Nr | awk '{print $2 "\ t" $1}'> baiduma. log

7. Extract the first 200 pages and the number of visits with a 100 Baidu spider access code, and sort them in descending order.

Cat *. log | grep 'baidider '| grep '000000' | awk' {print $5} '| sort | uniq-c | sort-Nr | head-N 200> baiduurl200.log

Or: grep '000000' Baidu. log | awk '{print $5}' | sort | uniq-c | sort-Nr | head-N 200> baidu200.log

8. Extract all abnormal status codes such as 404, 301, and 302 and sort them in descending order.

Awk '($11 ~ /404/) 'Baidu. log | awk' {print $11, $5} '| sort> baidu404.log

9. Calculate the number of unique URLs captured by the spider

Cat access. log | grep baiduspider + | awk '{print $7}' | sort-u | WC

Export: CAT access. log | grep baiduspider + | awk '{print $7}' | sort-u> baiduspiderurl.txt

The number of times each URL is crawled: CAT access. log | grep baiduspider + | awk '{print $7}' | sort | uniq-C> baiduspiderurl.txt

10. Capture the source IP address, time, captured URL, return code, and captured size of Baidu spider access

Grep "baiduspider +" example. log | awk '{print $1 "\ t" $4 "\ t" $7 "\ t" $8 "\ t" $9 "\ t" $10 }'> baiduspider.txt

11. The $ in the above command is customized here, and you can compare your own logs.