Shell--awk

1. awk command format:

awk [Options] File name

Options

-F FS: Specify delimiter for split data

-F File: Specifies the program's script file

Example: awk-f: ' {print '} '/etc/passwd

Awk–f script/etc/passwd

Script: Represents the processing statement for the passwd file

Can be placed in a script file like {print $}

2. BEGIN, END

Awk-f: ' Begin{num=0}{print $1;num++}end{print num} '/etc/passwd

Begin executes before the data is read, and end executes after reading the data.

Split the passwd file with ":" and then print the first element of each line.

3. Built-in variables

Fs:field seperator, field delimiter at input
# awk ' begin{fs= ': "}{print $1,$7} '/etc/passwd
Rs:record seperator, enter line delimiter
Ofs:output field Seperator, when the output is delimited;
Ors:outpput row seperator, the line delimiter at output;
Nf:numbers of field, number of fields
Nr:numbers of Record, number of rows; All documents are counted together;
FNR: number of rows; Each file is counted separately;

Usage:

awk ' {print FNR, ' $ ' gradt.txt line number

4. Matching characters

Cases:

awk '/110.52.250.126/{print '} ' access_2013_05_30.log all rows containing 110.52.250.126 elements

awk ' ($ ~/110.52.250.126/) {print '} ' Access_2013_05_30.log all the first element is a row of the 110.52.250.126 element

Awk ' ($!~/110.52.250.126/) {print '} ' Access_2013_05_30.log all the first element is not a row of the 110.52.250.126 element

5: Website Traffic log analysis

1): First remove some pictures, JS, CSS and some invalid Web requests

awk ' ($7!~/\.jpg|\.png|\.css|\.js|\.gif/&& $9 ~/200/) {print} ' Access_2013_05_30.log > clean_2013_05_30. Log

redirect filtered data to the Clean_2013_05_30.log log file

2): Statistics of total data in Clean_2013_05_30.log files (PV number)

awk ' begin{pv=0} {pv++} end{print PV} ' Clean_2013_05_30.log

3): Count the number of different IP accesses (number of Uvs)

awk ' {print '} ' access_2013_05_30.log|sort-n|uniq-c|wc-l

Note: You need to sort and then go back to this to be correct. Otherwise, the Uniq will only remove the equivalent of the next.

So first sort again go heavy and then in the statistics quantity

Note: awk ' {print '} ' access_2013_05_30.log|sort-n|head–10

You can use head to take out more traffic than the IP, see the number of visits to determine whether it is a crawler

Shell--awk