Getting Started with awk processing logs (GO)

Objective

These two days I rolled up my sleeves to deal with the logs and finally started to get awk. In fact, the basic use of awk, learning is a half-day time, before always rely on colleagues to do, lazy ah.

This article is only for beginners, ops do not onlookers.

The following is an example of the log being processed, not so standard, but the nonstandard log is the standard case.

[2015-08-20 10:00:55.600]-[192.168.0.73/192.168.0.75:1080 com.vip.xxx.myservice_2.0 0 106046 100346 90ms 110ms]

Basic statement

The most basic statement, separated by a space, extracts the required columns:

awk ' {print $0,$1,$2,$ (NF-1), $NF, $NF-$ (NF-1)} ' Access.log

1. Enter

AWK is the processing language that is entered for each line in a file or pipeline. So you can also enter from the pipeline:

grep "XXX" Access.log | awk ' {print '} '

But this would be the main character of the old Linux terrier, and awk doesn't need to be cat-less.

Cat Access.log | awk ' {print '} '

2. Statement definition

You can quickly use single quotes ' to write all the statements in one line.

You can also use the-F to specify the file, the file can be arbitrarily wrapped, increase readability and reusability.

All execution statements are enclosed in {}, and the outside of {} is something advanced such as filter conditions, see later.

3. Column references

$ A represents the entire row of data, representing the first column (not finally the number of programmers starting from 0).

NF is a system variable that represents the total number of columns, so $NF represents the last column and also supports $ (NF-1) to represent the penultimate column.

Operations between columns, such as $nf-$ (NF-1), are also supported for subtracting the values of the last two columns.

Write only one print, which is the abbreviation for print $, and prints the entire line of data.

4. Enter the column delimiter

The default is delimited by a space, can also be re-specified, the following example specifies ': '

Awk-f ': ' {print $1,$2} ' Access.log

You can also define multiple delimiters in a regular expression, and the following example specifies '-' and ': '

Awk-f ' [-:] ' {print $1,$2} ' Access.log

5. Column interval for output

Print $1,$2 The middle ', ' comma, which represents the use of the default delimiter between the 1th and 2nd columns when printing, which is a space, or you can use "" to define any other character:

awk ' {print ' \ t ' $ "-" $3$4xxxxx$5} ' Access.log

In the example above, the 1th 2nd column is Tab delimited, and the 2nd 3rd column is separated by "-",

can also do nothing to mean that there is no separation between the middle, such as 3rd 4th column, or write some characters useless "", also equals not written, such as 4th 5th column.

numeric type, String type

Although the last two columns of the previous example have a string type, they do not seem to be able to perform arithmetic operations with MS words.

But when you subtract two columns, awk magically converts them to pure numbers. Similarly, when you do accumulate, sum=sum+ $NF, and can be automatically converted to numbers.

If you want to compare a character column to a threshold, turn it back to the number first, and the previous article

Sed "s|ms]| | G "Access.log | awk ' $NF >100 {print} '

In fact, it can be as simple as the following, performance is slightly faster than using sed:

awk ' $NF *1>100 {print} ' Access.log

Or

awk ' int ($NF) >100 {print} ' Access.log

Begin and END statements

The statement after begin and end defines the statement before and after the full text content is processed.

1. Calculate the cumulative and average values

awk ' {sum+= $NF} END {print sum, sum/nr} '

The above example accumulates the value of the last column for each line of input, and the end statement prints the cumulative result and average, and NR is the system variable that represents the total number of rows.

2. Print the table header

You can also define a BEGIN statement to print the header, define variables, and so on.

awk ' Begin{print ' Date\t\ttime\t\tcost} {print $ \ t ' $ "\ T" $NF} ' Access.log

The previous table header is delimited with two tabs, and the contents are separated by a tab, which has good alignment effects.

Filter row 1. Simple character Matching

It is also possible to filter with grep first, or you can use awk to simply define regular expressions between//between execution statements

awk '/192.168.0.4[1-5]/{print '} ' Access.log

Equivalent to

grep "192.168.0.4[1-5]" access.log| awk ' {print $}

2. Match a character to a column

For the 4th column of the address segment match, ~ is a character match,!~ is the meaning of the mismatch.

awk ' $4 ~/192.168.0.4[1-5]/{print} '

3. Filtering for numeric values

Support = =,! =, <, <=;, >=

awk ' $ (NF-1) *1==100 {print} '

awk ' $NF-$ (NF-1) >100 {print} '

Before you see, for fields that are not pure numbers, you can use arithmetic operations to turn them back into numbers.

4. Multiple conditions exist simultaneously

awk ' (>150 | | $ (>250) {print} '

5. Using the IF statement

If the logic is more complex, consider using statements such as If,else

awk ' {if ($ (NF-1) *1>100) print} '

Other 1. External incoming parameters

For example, the threshold to pass in the timeout from outside, note the position of the threshold on the command line.

awk ' {if ($ (NF) *1>threshold) print} ' threshold=20 Access.log

2. Common functions

The most useful are gsub and sub,match,index. Where Gsub replaces a string with the target string, you can select an entire row to replace or just replace a column.

awk ' {gsub ("MS", "", $NF); if ($NF >100) print} ' Access.log

Some examples 1. Intercept segment data in a date segment

There are many ways to play freely with different log formats.

For example, the next intercept 17:30:30 seconds to 17.31:00 of the data, first extract the time division seconds three columns, and then spell a number to compare

Awk-f "[:.]" ' $2$3$4>=173030 && $2$3$4<173100 {print} '

You can also match an hourly time, the following example takes 11 points of the log:

awk '/[2015-08-20 11:/ {print "} ' Access.log

Take the data from 11:01 to 05 minutes:

awk '/[2015-08-20 11:0[1-5]:/ {print "} ' Access.log

2. Find out when the time-out data set occurred

The first paragraph finds out the timeout record, the second segment filters out the microseconds in the timestamp, and then merges by seconds, and counts the number of times that the second expires.

awk ' $ (NF) *1>100 {print} ' Access.log | Awk-f "." ' {print $} ' | Sort | Uniq-c

Original address: http://calvin1978.blogcn.com/articles/awk_accesslog.html

Getting Started with awk processing logs (GO)