Linux Operations Three Musketeers awk will knowledge-patterns and actions, inner arguments, statement blocks, arrays

Linux command The Three Musketeers awk
about awk
Awk is the boss of the Three Musketeers, and the power of awk is that it's not just a command, it's a language, it's very powerful. The report generator, which is displayed after formatting, can customize functions, and awk handles records and fields (domains).
How awk handles the data
Once the data is read, the data is fragmented first, by default, by a space delimiter, after the Shard is referenced to each shard by the location variable parameter, and then by the conditions of the awk command to print

Records and Fields
Before we talk about awk parameters, built-in variables, arrays, let's start with what is a record and what is a field? The field is well understood, that is, the default space is the partition after the Shard, each area is a field, just like the above is a field.
Record, this may be a bit around, for the time being, you can get a look at the built-in variables later, and then combine examples to understand. Awk handles objects that are records and fields (fields), rather than the lines that we often say, the text being read is a series of character records, and the end flag for each record is the line break "\ n", and this end mark can be changed, as you can see in the subsequent awk built-in variable.
awk Syntax
awk [Options] ' {print item} ' File1,file2 ... ' can also be the output of a command
"Item can be a positional parameter, such as $ $, or a string that needs to be caused by" ", and the item is separated from item with a", "comma"
The awk command consists of a pattern (pattern) and an action (action)

mode (pattern)
The pattern can be any one of the following:
1. Regular Expressions
When you use this regularization in awk: When you match its first character, you need to specify the parameter:--posix or--re-interval, otherwise you cannot use this pattern.

[[email protected] tmp]# cat awktest1one#two#three numzhangsan nanlisi nvwangwu nanlaowang nan[[email protected] tmp]# [[email protected] tmp]# awk ‘/^[^$]/‘ awktest1  ^在[]里面表示不匹配之意one#two#three numzhangsan nanlisi nvwangwu nanlaowang nan[[email protected] tmp]#[[email protected] tmp]# awk ‘/^lao/{print $2}‘ awktest1nan[[email protected] tmp]#

X{m}:x Repeat M times
X{m,}:x at least repeat m times
X{,m}:x up to repeat m times
X{m,n}:x repeats at least m times, but not more than n times

[[email protected] tmp]# cat awk1rootrrttgood[[email protected] tmp]# awk --posix ‘/o{2}/{print $0}‘ awk1rootgood[[email protected] tmp]#

2、关系表达式: Operation with operator, can be a string or numeric comparison test

, <, =, >=, <=

[[email protected] tmp]# awk ‘NR>=2{print $0}‘ awk1rrttgood[[email protected] tmp]#

3. Pattern matching Regular expression : with operator ~ (match) and ~ ~ does not match
[[email protected] ~]# ifconfig enp0s3 | awk -F "[: ]*" ‘$2~/inet$/ {print $3}‘
4, can also match the address range similar to SED

[[email protected] tmp]# cat awk1rootrrttgoodhellonihaonizaiganma[[email protected] tmp]#[[email protected] tmp]# awk ‘NR==2,NR==4{print $0}‘ awk1  rrttgoodhello[[email protected] tmp]#[[email protected] tmp]# awk ‘/^rr/,NR==3{print $0}‘ awk1 rrttgood[[email protected] tmp]#

5. BEGIN statement block, END statement block
Action (Action)
An action consists of one or more commands, functions, and expressions, separated by a newline or semicolon, and enclosed in curly braces, in the main part: variable or array assignment, Output command, built-in function, control-flow statement. For example: Print function, ' {print '} '
awk Command Processing flowchart
If the begin and end statement blocks are not defined, then the begin, end statement block is not executed.

awk Parameters
-F Re: Allows awk to change its field delimiter
-V var= $v assigns the V value to Var, if there are multiple variables to be assigned, then write multiple-V, each variable assignment corresponds to a-V
e.g. to print the line between the NUM line of file A and the Num+num1 line,
Awk-v num= $num-v num1= $num 1 ' nr==num,nr==num+num1{print} ' a
-F Progfile: Allows awk to invoke and execute the Progfile program file, of course Progfile must be a program file that conforms to the awk syntax.
The-f parameter is equivalent to "-V fs= the value to be assigned to FS", which specifies the field delimiter, such as-F ":" Equals-v fs=:, specifying a colon as the delimiter "
--posix or-re-interval: In some regular mode, you need to specify this parameter to use regular (f and--posix are commonly used)
awk built-in variables
$n: The nth field of the current record, for example, N is 1 for the first field, and N for 2 for the second field
$: This variable contains the text content of the current line during execution "whole line"
NR: Indicates the number of records, which corresponds to the current line number during execution
FS: Field delimiter (default is space)
OFS: Output field delimiter (default is space)
NF ($NF): Indicates the number of fields, corresponding to the current number of fields during execution; print $NF prints the last field in a row
ORS: Output record delimiter (default value is line break)
RS: Record delimiter (default is line break)
ARGC: Number of command-line arguments
Argind: The location of the current file on the command line (counting from 0)
ARGV: An array containing command-line arguments
CONVFMT: Number conversion format (default is%.6g)
ENVIRON: Environment variable associative array
ERRNO: Description of the last system error
FieldWidths: List of field widths (separated by space key)
FileName: The name of the current input file
FNR: with NR, but relative to the current file
IGNORECASE: If true, ignores case matching
OFMT: The output format of the number (the default value is%.6g)
Rstart: The first position of a string that matches a match function
Rlength: The length of a string matched by the match function
SUBSEP: Array subscript delimiter (default value is 34)
The Begin:awk command executes the operation before execution, where the variable can be defined (built-in variable)
After the End:awk command executes, it can be used to output results, statistics, empty lines, etc.
frequently used built-in variables:
NR,NF, $n, fs,ofs,rs,ors
Lab Environment:

[[email protected] tmp]# cat awktestHello 1      2      3 I am a teacher  4I like basketball  5

$n the nth field (positional variable) of the current record (row), with a space separator by default, and "$" for the entire row

[[email protected] tmp]# awk ‘{print $0}‘ awktestHello 1      2      3 I am a teacher  4I like basketball  5[[email protected] tmp]#[[email protected] tmp]# awk ‘{print $1}‘ awktest Hello23II[[email protected] tmp]#

NR: Indicates the number of input records (line number)

[[email protected] tmp]# awk ‘{print NR,$0}‘ awktest1 Hello 12       23       3 4 I am a teacher  45 I like basketball  5[[email protected] tmp]# awk ‘NR==4{print NR,$0}‘ awktest4 I am a teacher  4[[email protected] tmp]#

NF: Indicates the number of input fields per line
What is a field?
Example: Line 4th "I am a Teacher"
The default is a space delimiter, "I" The first field, "AM" the second field, and so on, the 4th line a total of 4 fields
"$NF" represents the last field of a row

[[email protected] tmp]# awk ‘{print NF,$0}‘ awktest2 Hello 11       21       3 5 I am a teacher  44 I like basketball  5[[email protected] tmp]# awk ‘{print $NF}‘ awktest 12345[[email protected] tmp]#

FS OR-F: Enter field delimiter, default to Space

[[email protected] tmp]# awk -F ":" ‘NR==1{print $2}‘ /etc/passwd x[[email protected] tmp]# awk ‘BEGIN{FS=":"}NR==1{print $2}‘ /etc/passwd         x[[email protected] tmp]#

OFS: Output field delimiter, default to Space
The default is not to change the result of the row, so you want to use the $ output, and want to change the results, then you need to give it an action in front, rarely use the output

[[email protected] tmp]# awk ‘BEGIN {FS=" ";OFS="@"}{print $0}‘ awktest   Hello 1      2      3 I am a teacher  4I like basketball  5 [[email protected] tmp]# awk ‘BEGIN {FS=" ";OFS="@"}$1=$1{print $0}‘ awktest            给个动作$0，使其改变输出结果[email protected]23[email protected]@[email protected]@4[email protected]@[email protected][[email protected] tmp]# awk ‘BEGIN {FS=" ";OFS="@"}{print $1,$2,$3}‘ awktest    [email protected]@[email protected]@[email protected]@[email protected]@a[email protected]@basketball[[email protected] tmp]#[[email protected] tmp]# awk ‘BEGIN{FS="[:/]";OFS="#"}NR==1{print $1,$NF}‘ /etc/passwdroot#bash[[email protected] tmp]#

RS: Enter a record line delimiter, default to line break (read-in delimiter, meaning a symbol as a line delimiter to divide rows, default is newline character \ n)
Awk handles objects that are records and fields (fields), rather than the lines that we often say, the text being read is a series of character records, and the end flag for each record is the newline character "\ n".

[[email protected] tmp]# awk ‘{print NR,$0}‘ awk.txt 1 root:x:0:0:root:/root:/bin/bash\n2 bin:x:1:1:bin:/bin:/sbin/nologin\n[[email protected] tmp]# awk ‘BEGIN{RS="/"}{pritn NR,$0}‘ awk.txt            现在把划分行的分隔符改为“/”，存在“/”就是回车换行awk: BEGIN{RS="/"}{pritn NR,$0}awk:                       ^ syntax error[[email protected] tmp]# awk ‘BEGIN{RS="/"}{print NR,$0}‘ awk.txt   1 root:x:0:0:root:2 root:3 bin4 bashbin:x:1:1:bin: 因为\n已经不是记录分隔符，所以这行不会被算作记录行5 bin:6 sbin7 nologin        因为\n已经不是记录分隔符，所以这行不会被算作记录行[[email protected] tmp]#[[email protected] tmp]# cat ricky 1\n2\n3\n1\n4\n5\n1\n6\n7\n[[email protected] tmp]# awk ‘BEGIN{RS="1"}{print $1,$2,$3}‘ ricky     1作为分隔符，所空行2 3 \n不再是分隔符，所以2与3之间的\n，成为了一个空格存在4 5 6 7 [[email protected] tmp]#

ORS: Output record line delimiter, default to line feed (delimiter at output), reverse operation of RS
Turn the following text into a line

one\ntwo\nthree\n[[email protected] tmp]# awk ‘BEGIN{ORS=" "}{print $0}‘ awktest2 （把换行符"\n"转变为空格）   

BEGIN statement Block
This is done before the awk command executes, where variables can be defined (built-in variables)

[[email protected] tmp]# awk ‘BEGIN{FS=":"}NR==1{print $2}‘ /etc/passwd         x[[email protected] tmp]#

End Statement block
The awk command executes after execution, and can be used to output results, statistics, empty lines, and so on.
Example 1:
Number of empty lines in the statistics file
grep implementation

[[email protected] tmp]# grep -c "^$" /etc/init.d/sshd （-c计算符合条件的行数）20 [[email protected] tmp]#

awk implementation

[[email protected] tmp]# awk ‘/^$/{a++;print a}‘ /etc/init.d/sshd （调试，每次匹配都输出a的值） 1234567891011121314151617181920[[email protected] tmp]# awk ‘/^$/{a++}END{print a}‘ /etc/init.d/sshd(统计信息，在awk命令结束后输出a的值，“/^$/{a++}，条件匹配一次，变量a的值加1”)  20[[email protected] tmp]#

Example 2:
Find the variables in the environment $path, all only three arbitrary characters of the command, such as tee, and say they redirect to Command.txt, ask for a row to display one, and at the end of the file to count their number.

[[email protected] tmp]# echo $PATH/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/ Bin:/root/bin[[email protected] tmp]# echo $PATH | TR ":" \ n "/usr/local/sbin/usr/local/bin/sbin/bin/usr/sbin/usr/bin/root/bin/bin/root/bin[[email protected] tmp]# Find $ (echo $PATH | tr ":" \ n ")-type f-name"??? " 2>/dev/null |awk ' {a++}{print $0}end{print ' result: ' A} ' >>command.txt (? In the command, the number of wildcard characters, any one character;?) The wildcard user matches the file name, whereas in the regular matches its first character 0 or 1 times) [[email protected] tmp]# cat command.txt/sbin/cbq/sbin/lvm/sbin/sln/sbin/arp/ bin/env/bin/raw/bin/cat/bin/cut/bin/rpm/bin/sed/bin/tar/bin/pwd/usr/sbin/lid/usr/sbin/zic/usr/bin/lua/usr/bin/ grn/usr/bin/dir/usr/bin/who/usr/bin/tac/usr/bin/seq/usr/bin/xxd/usr/bin/cpp/usr/bin/cmp/usr/bin/idn/usr/bin/ fmt/usr/bin/yum/usr/bin/pic/usr/bin/tic/usr/bin/eqn/usr/bin/toe/usr/bin/man/usr/bin/tee/usr/bin/ptx/usr/bin/ ldd/usr/bin/c89/usr/bin/s2p/usr/bin/a2p/usr/bin/sum/usr/bin/sar/usr/bin/vim/usr/bin/top/usr/bin/tbl/usr/bin/ Tty/usr/bin/c99/usr/bin/gcc/usr/bin/rev/usr/bin/col/usr/bin/yes/usr/bin/cal/bin/env/bin/raw/bin/cat/bin/cut/bin/rpm/bin/sed/ Bin/tar/bin/pwdresult:57[[email protected] tmp]#

Array
Structure of the array

Process the following file contents, remove the domain name and sort the statistics according to the domain name
Http://www.baidu.com/index.html
Http://www.baidu.com/1.html
Http://post.baidu.com/index.html
Http://mp3.baidu.com/index.html
Http://www.baidu.com/3.html
Http://post.baidu.com/2.html

Programme one:

[[email protected] tmp]# awk ‘BEGIN{FS="/"}{print $3}‘ baidu.txt | sort | uniq -c      1 mp3.baidu.com      2 post.baidu.com      3 www.baidu.com[[email protected] tmp]#

Scenario Two:

[[email protected] tmp]# awk ‘BEGIN{FS="/"}{array[$3]++}END{for (key in array)print key,array[key]}‘ baidu.txt post.baidu.com 2www.baidu.com 3mp3.baidu.com 1[[email protected] tmp]#

Parsing arrays
array[$3]++ > Array[$3]= array[$3]+1=0+1 (array[$3] Initial value is 0)
array[www.baidu.com]++ >0+1 1
array[www.baidu.com]++ >1+1 2
array[post.baidu.com]++ >0+1 1
Array[mp3.baidu.com (array name)]++ >0+1 1 (the value of the array)
array[www.baidu.com]++ >2+1 3
array[post.baidu.com]++ >1+1 2

For key in array (representing array name)
For key in post.baidu.com www.baidu.com mp3.baidu.com

Array[key]
array[post.baidu.com]++ >1+1 2
array[www.baidu.com]++ >2+1 3
array[mp3.baidu.com]++ >0+1 1

Summarize
Records and Fields
Pattern: regular; pattern matching regular; sed address range; relationship expression
Operation
Parameters:--posix or-re-interval
Built-in variables: FS;OFS;RS;ORS;NF ($NF); $n ($)
Statement BLOCK: Begin;end
Array

Linux Operations Three Musketeers awk will knowledge-patterns and actions, inner arguments, statement blocks, arrays