Linux command-awk Getting Started

Most recently, I often look at nginx logs, sometimes need to do some statistical analysis, so I think of awk, learning to make a record.

Directory

• Overview: A brief introduction to the awk background principle
• Basic usage: Commonly used awk syntax
• Built-in variables
• Comprehensive examples

Overview

Awk is the founder of Aho, Kernighan and Weinberger, a column-based streaming tool that is functionally a combination of sed and grep, but not only that, awk itself is a programming language. Awk reads each row sequentially (by dividing by row by default, you can specify the delimiter for the record), and then divides the row into several fields based on the delimiter (the default is blank, one or more spaces, tab), and each field is named $1,$2...,$0 to represent a whole row

Basic usage

Awk can be used either directly at the command line or as a script (when using complex statements)

[-F value]  ' Program Text ' [File ...] # using the script file awk [-F value]  [-F Program-file] [--] [file ...]

Use the command line directly because the temporary application is relatively simple.

• ' Program text ': Represents the awk statement in the form "Pattern { ACTIONS; } ", where patern is the regular expression, and action is the action for the previous matching text, such as the print
• -F: It says that the default delimiter for awk is blank, using the-F option to specify delimiters, such as using "," as a delimiter

awk -F,  '{print}' awk.txt

For awk segmentation methods, such as text awk.txt

John.wang Male 30 021-111111
Lucy.yang Female 25 021-222222
Jack.chen Male 35 021-333333
Lily.gong Demale 021-444444 Shanghai

Run awk ' {print $} ' Awk.txt

John.wanglucy.yangjack.chenlily.gong

Awk first reads the first line, divides it into 4 fields, and the first row is the entire row, as follows

John.wang       male          021-111111   $1             $2    $3          $4

So the above output is the first column.

Print and printf

The first is that these two are not functions in C, but Awk's built-in commands

Print: Wrap after printing, there is no interval between multiple values printed

printf: Richer formatted output, printing one line cannot wrap

Built-in variables

AWK provides some built-in variables as a programming language, here's a look at common

NF        current record number of Colum nr of the        current record, if row by line is the line number        of Rs per record of the separator, the default is "\ n" 

NF: Sometimes we don't know how many columns each record has, but we want to print out the last column so we can use {print $NF}

NR: Sometimes we just want to work with some lines in the text, you can use NR to specify the rows to be processed

Comprehensive application

Ready so much to finally start analyzing the log, my log format is as follows

5x.247.204. -- [ A/jul/ .: -: -: -+0800]"Post/user/login http/1.1"  404 178  "Http://www.xxxx.com/home/views/index/index"  "mozilla/5.0 (Windows NT 6.1; trident/7.0; rv:11.0) Like Gecko"1x.153.105.159- [ A/jul/ .: -: -: -+0800]"post/user/send http/1.1"  404 178  "Http://www.xxxx.com/home/views/index/index"  "mozilla/5.0 (Windows NT 6.1; trident/7.0; rv:11.0) Like Gecko"1x.153.105.161- [ A/jul/ .: -: -: -+0800]"post/user/regist http/1.1"  503 222  "Http://www.xxxx.com/home/views/index/index"  "mozilla/5.0 (Windows NT 6.1; trident/7.0; rv:11.0) Like Gecko" 

Because I received a lot of malicious access, I wanted to count the number of IP accesses to the interface /user/login , starting with SED and awk

sed ' 10000,${/user\/login/p} ' awk ' {print $} ' | Sort Uniq Sort -N

• -N: Select line number
• '10000,${/user\/login/p}': Starting with 10000 lines, comma "," followed by no value, represented to the last line, looking for a matching "/user\/login" line, p means print out
• awk ' {print $} ': Print out the first column (i.e. IP)
• Sort: Sorting the output IP using sort
• Uniq-c: de-consolidating statistics for sorted IPs (general and sort, because Uniq can only merge consecutive identical rows)
• Sort-n: Sort the output by a number

The output is as follows (for IP processing using "x" coding):

Wuyi 1xx.41. 12.17  the 1xx.190. 94.49  Wu 2xx.4. 47.166  About 2xx.203. 63.178  - 1xx.62. 101.177  - 2xx.133. 116.25

In this way, the illegal access IP can be statistically masked using Nginx's blacklist function.
In fact, the role of SED is to limit the specified line and search, in fact, awk itself can do

awk ' nr>=760&&nr<=890 {print $} ' Sort | Uniq Sort -N

The built-in variable nr is used above, and the awk logic operation && is used, and the specific syntax of awk is frequently studied later, otherwise it will not be forgotten without learning.

Linux command-awk Getting Started