Reading Notes-Common commands for online log analysis and Common commands for log analysis

Reading Notes-Common commands for online log analysis and Common commands for log analysis

From large-scale distributed website architecture design and practice 4.1 online Log Analysis

1. Basic commands

Operation	Command	Description
View File Content	Cat-n access. log	-N: display the row number
Display files by PAGE	More access. log	Enter next line, space next page, F next screen, B Previous Screen
Display files by PAGE	Less access. log	Search and highlight input/string
Show end of File	Tail-n2-f access. log	-N2: The last two rows are displayed.-f continues to listen and does not quit.
Content sorting	Sort-k 2-t''-n access. log	-K specifies the sorting column,-t specifies the column separator, and-n is in numerical order.
Character statistics	Wc-l access. log	-L number of statistical lines,-c characters,-L maximum line length,-w words
View duplicate rows	Sort testfile | uniq-c-d	Uniq deduplication,-c counts the number of duplicates, and-d only displays duplicates
String search	Grep 'G. * t' access. log	Search for strings starting with G and ending with T
File Search	Find/home/java-name access. log	Recursive/home/java subdirectory named access. log
Evaluate expressions	Expr 10 \ * 3	Calculate 10*3, where \ * indicates that escape * does not read as a wildcard
Evaluate expressions	Expr length "this is a test"	Length
Archive files	Tar-cf aaa.tar f1 f2	-C creation,-f specifies the package name
Archive files	Tar-xf aaa.tar	-X Extract
URL access	Curl www.google.com	The response body is returned without parameters.-I returns the response header, and-I returns only the response header.
View the CPU load	Uptime
View CPU usage	Top | grep Cpu	View each core by 1, and view by shift + H by thread
View CPU usage	Top-p 2864	View a specified process
Remaining disk space	Df-h
Remaining disk space	Du-d 1-h/home/java	Disk usage of the analysis directory. -D: Set recursive depth.
Network traffic	Sar-n DEV 1 1	View each network card in DEV. sampling is performed once every 1 second.
Disk I/O	Iostat-d-k
Memory usage	Free-m	Generally, the used and free values corresponding to "-/+ buffers/cache" are used.
Memory usage	Vmstat	View swap I/O

 

2. sed Editor

Operation	Command	Description
Text replacement	Sed's/xxx/yahoo/'access. log | head-10	The original file is not modified.
Output the specified row	Sed-n'2, 6p' access. log	Output 2-6 rows
Delete row	Sed '/qq/d' access. log	/D exclude rows containing keywords qq
Full Line replacement	Sed-E'/google/c \ hello' access. log | head-10	/C replaces the entire matched row.
Merge multiple commands	Sed-n '1, 5 p; = 'access. log	Print row and print row number

 

3. awk Program

Operation	Command
Filter rows and print specified Columns	Awk '/google/{print $0, $6}' access. log | head-10
Print by conditions	Awk 'length ($0)> 40 {print $3} 'access. log | head-10
Format output	Awk '{line = sprintf ("method: % s, response: % s", $3, $7); print line}' access. log | head-10

 

4. Example

Operation	Command	Description
View request traffic
Top 10 IP addresses	Cat access. log | cut-f1-d "" | sort | uniq-c | sort-k 1-n-r | head-10	The cut part indicates that the 1st column is the IP column, and the 4th column is the URL access traffic.
View the most time-consuming page	Cat access. log | sort-k 2-n-r | head-10	Sort by 2nd-column response time in reverse order
Count the percentage of 404 requests
Total requests	Export total_line = 'wc-l access. log | cut-f1-d ""'
404 requests	Export not_found_line = 'awk' $6 = '000000' {print $6} 'access. log | wc-l'
Proportion	Expr $ not_found_line \ * 100/$ total_line	& Or; can be merged into one row.