Linux command: Introduction to Awk

Awk is a powerful text analysis tool, with the search for grep and the editing of SED, which is especially powerful when it comes to analyzing data and generating reports. To put it simply, awk reads the file line-by-row, using spaces as the default delimiter to slice each row, and then perform various analytical processing of the cut.

Note: awk has 3 different versions: AWK, Nawk, and gawk, which are not specifically described, generally referred to as the GNU version of awk, Gawk,gawk.

Grammar:

awk [Options] ' program ' File File ...

awk [Options] ' pattern{action} ' file File ...

-F CHAR: Specify delimiter

The output of awk:

Print Item1, item2,...

Points:

(1) Each item is separated by commas, while output is separated by an output delimiter

[[email protected] ~]# awk-f ":" '/^root/{print $1,$7} '/etc/passwd

Root/bin/bash

(2) Each item of the output can be a string or numeric value, a field of the current record, a variable, or an expression of awk, and the value will be implicitly converted to a string after the output

[[email protected] ~]# cat/etc/passwd |awk-f ': ' BEGIN {print ' name Shell '} {print $1,$7} '

Name Shell

Root/bin/bash

Bin/sbin/nologin

Daemon/sbin/nologin

Adm/sbin/nologin

Lp/sbin/nologin

(3) Print after item if omitted, equivalent to print $; output blank, use Pirnt ""

[[email protected] ~]# awk-f ":" '/^root/{print} '/etc/passwd

Root:x:0:0:root:/root:/bin/bash

[[email protected] ~]# awk-f ":" '/^root/{print $} '/etc/passwd

Root:x:0:0:root:/root:/bin/bash

[[email protected] ~]# awk-f ":" '/^root/{print ""} '/etc/passwd

Variables for awk:

Built-in variables, custom variables

(1) Built-in variables:

Fs:field seperator, Field delimiter at input (equivalent to-f:)

Rs:record seperator, enter line delimiter

Ofs:output field seperator, Fields delimiter at output

Ors:outpput row seperator, line delimiter at output

Nf:numbers of field, number of fields

Nr:numbers of Record, number of rows; count all files together

FNR: Number of rows; count each file separately

ARGV: An array that holds the character of the command itself, awk ' {print $} ' 1.txt 2.txt, meaning argv[0] saves awk,

ARGC: Save the number of arguments in the awk command

Filename:awk the name of the current file being processed

Example:

Specifying delimiters using FS

[[email protected] ~]# awk ' begin{fs= ":"} {print $1,$7} '/etc/passwd

Root/bin/bash

Bin/sbin/nologin

Daemon/sbin/nologin

Adm/sbin/nologin

Lp/sbin/nologin

Sync/bin/sync

Statistics/etc/passwd: File name, line number per line, number of columns per row, corresponding full line contents

[Email protected] ~]# awk-f ":" ' {print ' filename: "filename", linenumber: "NR", Columns: "NF", Linecontent: "$"/etc/ passwd

Filename:/etc/passwd,linenumber:1,columns:7,linecontent:root:x:0:0:root:/root:/bin/bash

Filename:/etc/passwd,linenumber:2,columns:7,linecontent:bin:x:1:1:bin:/bin:/sbin/nologin

Custom variables

-V Var_name=value

Variable name Distinguishing character case

(1) variable can be defined in program

(2) You can customize variables by using the-v option in the command line

Example: Statistics on the number of/ETC/PASSWD accounts

[Email protected] ~]# awk-f: ' {count++;p rint $} end{print "User Count is", count} '/etc/passwd

Root:x:0:0:root:/root:/bin/bash

Bin:x:1:1:bin:/bin:/sbin/nologin

Daemon:x:2:2:daemon:/sbin:/sbin/nologin

Adm:x:3:4:adm:/var/adm:/sbin/nologin

Lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

User Count is 5

printf command

Use format for commands: printf format, item1, item2,...

Points:

(1) to specify format;

(2) will not be automatically wrapped, if you want to change the line, you need to give \ n

(3) format is used to specify its output format for each subsequent item;

The format indicator is preceded by%, followed by a character:

%c: The ASCII code that displays the characters;

%d,%i: decimal integer;

%e,%e: The scientific counting method shows the numerical value;

%f: Displays floating-point numbers;

%g,%g: Displays values in scientific notation format or floating-point number format;

%s: Display string;

%u: Displays unsigned integers;

Percent: show% itself;

Modifier:

#: Display width

-: Align Left

+: Display symbols for numeric values

. #: Value Accuracy

Example: Displaying the UID of a user

[[email protected] ~]# awk-f: ' {printf ' UID:%d\n ", $ $} '/etc/passwd

uid:0

Uid:1

Uid:2

Uid:3

Uid:4

Uid:5

Uid:6

Uid:7

Uid:8

Statistics/etc/passwd: File name, line number per line, number of columns per row, corresponding full line contents

[Email protected] ~]# awk-f ': ' {printf ("filename:%10s,linenumber:%s,columns:%s,linecontent:%s\n", FILENAME,NR,NF, $)} '/etc/passwd

Filename:/etc/passwd,linenumber:1,columns:7,linecontent:root:x:0:0:root:/root:/bin/bash

Filename:/etc/passwd,linenumber:2,columns:7,linecontent:bin:x:1:1:bin:/bin:/sbin/nologin

Awk's operator

Arithmetic operators:

X+y

X-y

X*y

X/Y

X**y, X^y

X%y

-X: Negative value

+x: Convert to Numeric

String operators: Connecting

Assignment operators:

=

+=

-=

*=

/=

%=

^=

**=

++

--

If the pattern itself is the = sign, write as/=/

Comparison operators:

<

<=

>

>=

==

!=

~: Pattern matching, the left string can be matched to the right pattern to true, otherwise false;

!~:

Logical operators:

&&: With

|| : OR

Conditional expression:

Selector?if-true-expression:if-false-expression

Example: Viewing a user with a UID greater than 500 displayed as a normal user, otherwise displayed as an administrator or system user

[Email protected] ~]# awk-f: ' {$3>500?utype= ' Common user ': utype= "Admin or Syetem user";p rint $, "is", Utype} '/etc/p asswd

Root is Admin or Syetem user

Bin is Admin or Syetem user

Daemon is Admin or Syetem user

ADM is Admin or Syetem user

User9 is Common User

User10 is Common User

Mode:

(1) Regexp: Format is/pattern/

Only rows that are matched to by/pattern/are processed

(2) Expression: expression that satisfies the condition if the result is not a 0 or a non-empty string

Only the rows that meet the criteria are processed

(3) Ranges: Line range, previous address delimitation

NR only handles rows in the range

(4) Begin/end: A special mode that executes only once before the program of the awk command runs (BEGIN) or after run (END)

(5) Empty: null mode, matches any row

Example: Displaying the user name and shell type

[Email protected] ~]# awk-f: ' $NF ~/bash$/{print $, $NF} '/etc/passwd

Root/bin/bash

Mk/bin/bash

Bash/bin/bash

Testbash/bin/bash

Basher/bin/bash

Displays user names and shell types within a specified range

[Email protected] ~]# awk-f: ' Nr>=1&&nr<=5{print $, $NF} '/etc/passwd

Root/bin/bash

Bin/sbin/nologin

Daemon/sbin/nologin

Adm/sbin/nologin

Lp/sbin/nologin

Count the number of bytes occupied by a file under a folder

[[email protected] ~]# ll/etc/|awk ' BEGIN {size=0} {size=size+$5} end{print ' [End]size is ', size} '

[End]size is 1995496

Count the number of bytes in a file under a folder, displayed in M

[[email protected] ~]# ll/etc/|awk ' BEGIN {size=0} {size=size+$5} end{print "[End]size is", size/1024/1024, "M"} '

[End]size is 1.90305 M

Common action:

(1) Expressions

(2) Control statements

(3) Compound statements

(4) Input statements

(5) Output statements

Control statements:

If-else

Format: if (condition) {then body} else {else body}

Determine the user UID size and display the user's type

[Email protected] ~]# awk-f: ' {if ($3>=500) {print $ ' is a Common User '} else {print $ ' is a Admin or System User} } '/etc/passwd

Rootis a Admin or System User

Binis a Admin or System User

User9is a Common User

User10is a Common User

Determine if the user UID is the same as the GID

[[email protected] ~]# awk-f: ' {if ($3=$4) {print $ "is a good guy"} else {print $ "is a bad guy"}} '/etc/passwd

Rootis a bad guy

Binis a good guy

Daemonis a good guy

Admis a good guy

Lpis a good guy

Syncis a bad guy

While

Format: while (condition) {while body}

Show odd lines of files

[[email protected] ~]# awk ' {i=1;while (I<=NF) {printf "%s", $i; i+=2};p rint ""} '/etc/inittab

#isusedupstarttherunlevel.

#

#OTHERHEREHAVEEFFECTYOUR

#

#initializationstarted/etc/init/rcs.conf

Do-while Cycle

Format: do {Do-while body} while (condition)

For loop

Format: for (variable assignment; condition, iteration process) {for body}

Example: View the number of accesses per IP that accesses a Web server

[email protected] ~]# ab-n 10000-c http://192.168.111.128/index.html (Website Access stress test)

This is apachebench, Version 2.3 < $Revision: 655654 $>

Copyright 1996 Adam Twiss, Zeus technology LTD, http://www.zeustech.net/

Licensed to the Apache software Foundation, http://www.apache.org/

Benchmarking 192.168.111.128 (Be patient)

Completed requests

Completed requests

[[email protected] ~]# awk ' {ip[$1]++}end{for (i in IP) {print i,ip[i]}} '/var/log/httpd/access_log

192.168.111.128 30031

Next

End the processing of the bank in advance and proceed to the next line;

Example: printing an odd number of users

[Email protected] ~]# awk-f: ' {if ($3%2==0) next;print $1,$3} '/etc/passwd

Bin 1

ADM 3

Sync 5

[Email protected] ~]# awk-f: ' {if (nr%2==0) next;print nr,$1} '/etc/passwd

1 root

3 Daemon

5 LP

Array

Traditional arrays: The index number starts at 1;

Associative arrays:

Array[index-expression]

Index-expression: Any string can be used; If an array element does not exist beforehand, awk automatically creates the element and initializes it to an empty string when referenced, so that the "index in array" format must be used to determine if an array exists;

a[first]= "Hello awk"

Print A[second]

To enumerate through each element of the group, you need to use a special structure like this:

For (var. array) {for body}

Its var iterates through the index of the array;

state[listen]++

state[established]++

Example: Show the number of link states such as listen or established

[Email protected] ~]# Netstat-tan | awk '/^tcp/{++state[$NF]}end{for (s in state) {print S,state[s]}} '

Established 1

LISTEN 11

Example: View the number of accesses per IP that accesses a Web server

[email protected] ~]# ab-n 10000-c http://192.168.111.128/index.html (website stress test)

This is apachebench, Version 2.3 < $Revision: 655654 $>

Copyright 1996 Adam Twiss, Zeus technology LTD, http://www.zeustech.net/

Licensed to the Apache software Foundation, http://www.apache.org/

Benchmarking 192.168.111.128 (Be patient)

Completed requests

Completed requests

[[email protected] ~]# awk ' {ip[$1]++}end{for (i in IP) {print i,ip[i]}} '/var/log/httpd/access_log

192.168.111.128 30031

To delete an array element:

Delete Array[index]

Awk's built-in functions

Split (String,array[,fieldsep[,seps]):

Function: The string represented by FIELDSEP is sliced as a delimiter, and the result of the slice is saved to an array with the array name, and the subscript is starting from 1;

Root:x:0:0::/root:/bin/bash

user[1]= "Root", user[2]

This function has a return value, and the return value is the number of elements after the slice

[Email protected] ~]# Netstat-tn | awk '/^tcp/{lens=split ($5,client, ":"); ip[client[1]]++}end{for (i in IP) print I,ip[i]} '

192.168.111.1 1

Length (String)

Function: Returns the length of a given string

SUBSTR (String,start[,length])

Function: Take a substring from a string, from start to the starting position as a substring of length;

Linux command: Introduction to Awk