Awk II, AWK advanced

awk command

awk [Options] '/pattern{action} '

Options-f

Work Traversal mode: iteration, loop

Pattern

Delimitation: ADDR1,ADDR2

Expression: $ > 500

/pattern/mode

begin{}: Executes once before the traversal operation begins

end{}: After the traversal operation ends, the command exits the self-signed execution once

ACTION:PRINT,PRINTF: For tools that enable formatted output

AWK programming Language:

variables, arrays

Select, Cycle

Built-in functions

Custom functions

The output of awk:

First, print

Use format for print:

Print item1, item2, ...

Points:

1, the items are separated by commas, and the output is separated by a blank character;

2. The output item can be a string or numeric value, a field of the current record (such as $ $), a variable, or an expression of awk, and the value will be converted to a string before output;

3, the Print command after the item can be omitted, at this time its function is equivalent to print $, so if you want to output blank lines, you need to use print "";

Example:

# awk ' BEGIN {print ' line one\nline two\nline three '} '

Awk-f: ' {print $, $/etc/passwd} '

Second, awk variables

2.1 awk built-in variable's record variable:

Fs:field separator, the field delimiter used when reading the file;

Rs:record separator, enter the text information using the line break;

Ofs:output Filed Separator:

Ors:output Row Separator:

Awk-f:

ofs= "#"

Fs= ":"

2.2 The data variables for awk built-in variables:

Nr:the number of the input Records,awk command, and if there are multiple files, this number will uniformly count the rows of the processed multiple files;

Nf:number of field, current record number of field;

FNR: Unlike NR, FNR is used to record the number of rows being processed that are currently processed in this file in total;

ARGV: Array, save the command line itself this string, such as awk ' {print $} ' a.txt b.txt This command, argv[0] Save awk,argv[1] Save a.txt;

The number of arguments to the Argc:awk command;

The name of the file that the Filename:awk command handles; Gets the current file name in the command

ENVIRON: An associative array of current shell environment variables and their values;

such as: awk ' begin{print environ[' PATH '} '

Make calculator using awk BEGIN {print +}

2.3 User-defined variables

Gawk allows users to customize their own variables for use in program code, where variable name naming rules are the same as most programming languages, with only letters, numbers, and underscores, and cannot start with a number. The gawk variable name distinguishes the case of characters.

-V: Custom variable

-V num1=20-v num2=32

2.3.1 Assigning variables in a script

Assign values to variables in gawk using assignment statements, for example:

awk ' begin{var= ' variable testing ";p rint var} '

2.3.2 using assignment variables in the command line

The Gawk command can also assign a value to a variable outside of the script and reference it in the script. For example, the above example can also be rewritten as:

AWK-V var= "Variable testing" ' Begin{print var} '

Third, printf

Use format for printf commands:

printf format, item1, ITEM2, ...

Points:

1. The biggest difference from the print command is that printf needs to specify format;

2. Format is used to specify the output format of each subsequent item;

3. The printf statement does not automatically print line breaks; \ n

The format indicator is preceded by a%, followed by a character, as follows:

%c: The ASCII code that displays the characters;

%d,%i: decimal integer;

%e,%e: The scientific counting method shows the numerical value;

%f: Displays floating-point numbers;

%g,%g: Displays values in the format of scientific notation or in the format of floating-point numbers;

%s: Display string;

%u: unsigned integer;

Percent: show% itself;

Modifier:

N: Display width;

-: left-aligned;

+: Display numerical symbols;

Example:

# awk-f: ' {printf '%-15s%i\n ', $1,$3} '/etc/passwd

Iv. output Redirection

Print Items > Output-file

Print Items >> output-file

Print Items | Command

Special File Descriptor:

/dev/stdin: Standard input

/dev/sdtout: Standard Output

/dev/stderr: Error Output

/dev/fd/n: A specific file descriptor, such as/dev/stdin, is equivalent to/dev/fd/0;

Example:

# awk-f: ' {printf '%-15s%i\n ', $1,$3 > '/dev/stderr '} '/etc/passwd

Vi. operator of awk:

6.1 Arithmetic operators:

-X: Negative value

+x: converted to numerical value;

X^y:

X**y: The second party

X*y: Multiplication

X/Y: Division

X+y:

X-y:

X%y:

6.2 String Operators:

There is only one, and you do not have to write it, to implement string connections;

6.3 Assignment operators:

=

+=

-=

*=

/=

%=

^=

**=

++

--

It should be noted that if a pattern is = number, there may be grammatical errors when using/=/, and should be replaced by/[=]/;

6.4 Boolean value

In awk, any non-0 value or non-empty string is true, whereas the other is false;

6.5 Comparison operators:

x < yTrue if x is less than Y.

X <= yTrue if x is less than or equal to Y.

X > YTrue if x is greater than Y.

X >= yTrue if x is greater than or equal to Y.

x = =y True if x is equal to Y.

X! =y True if x is not equal to Y.

x ~ yTrue If the string x matches the regexp denoted by Y.

X!~ yTrue If the string x does not match the regexp denoted by Y.

Subscript in array True if the array array has a element with the subscript subscript.

6.7 Logical relationships between expressions:

&&

||

6.8-Bar expression:

Selector?if-true-exp:if-false-exp

If selector; Then

If-true-exp

Else

If-false-exp

Fi

A=3

B=4

A>b?a is max:b ia max

6.9 function Calls:

Function_name (PARA1,PARA2)

The Seven awk model:

awk ' program ' Input-file1 input-file2 ...

The program is:

Pattern {Action}

Pattern {Action}

...

7.1 Common pattern types:

1, Regexp: Regular expression, the format is/regular expression/

2, expresssion: expression, whose value is not 0 or a non-null character satisfies the condition, such as: $ ~/foo/or $ = = "Magedu", with operator ~ (match) and!~ (mismatch).

3, Ranges: Specify the matching range, the format is PAT1,PAT2

4. Begin/end: Special mode, run only once or before the end of the awk command execution

5, empty (null mode): match any input line;

7.2 Common action

1, Expressions:

2. Control statements Combination

3, Compound statements

4. Input statements

5. Output statements

/Regular expression/: An extension set that uses wildcard characters.

Relational expressions: You can use the relational operator in the following operator table, which can be a string or numeric comparison, such as $2>%1 to select a row with a second field that is longer than the first word.

Pattern-Matching Expressions:

Mode, Mode: Specifies the range of a row. The syntax cannot include the begin and end patterns.

BEGIN: Lets the user specify the action that occurs before the first input record is processed, which is where the global variable is usually set.

End: The action that occurs after the last input record has been read by the user.

Eight control statements:

8.1 If-else

Syntax: if (condition) {Then-body} else {[Else-body]}

# awk ' {if ($3==0) {print $, ' adminitrator ';} else {print $, ' Common User '} '/etc/passwd

Example:

Awk-f: ' {if ($1== "root") print $, "Admin", else print $, "Common User"} '/etc/passwd

Awk-f: ' {if ($1== "root") printf "%-15s:%s\n", $, "Admin", Else printf "%-15s:%s\n", $, "Common User"} '/etc/passwd

Awk-f:-V sum=0 ' {if ($3>=500) sum++}end{print sum} '/etc/passwd

8.2 While

Syntax: while (condition) {statement1; statment2; ...}

Awk-f: ' {i=1;while (i<=3) {print $i; i++}} '/etc/passwd

Awk-f: ' {i=1;while (I<=NF) {if (length ($i) >=4) {print $i}; i++}} '/etc/passwd

awk ' {i=1;while (I<=NF) {if ($i >=100) print $i; i++}} ' Hello.txt

The contents of the Hello.txt file are a bunch of random numbers.

8.3 Do-while performs at least one loop body, regardless of condition

Syntax: do {statement1, Statement2, ...} while (condition)

Awk-f: ' {i=1;do {print $i; I++}while (i<=3)} '/etc/passwd

Awk-f: ' {i=4;do {print $i; I--}while (i>4)} '/etc/passwd

8.4 for

Syntax: for (variable assignment; condition; iteration process) {statement1, Statement2, ...}

Awk-f: ' {for (i=1;i<=3;i++) print $i} '/etc/passwd

Awk-f: ' {for (i=1;i<=nf;i++) {if (length ($i) >=4) {print $i}}} '/etc/passwd

The For loop can also be used to iterate over an array element:

Syntax: for (i in array) {statement1, Statement2, ...}

Awk-f: ' $NF!~/^$/{bash[$NF]++}end{for (A in BASH) {printf "%15s:%i\n", A,bash[a]}} '/etc/passwd

bash[/bin/bash]++

Bash[/sbin/nologin]

Bash[/bin/bash]=1

bash[/sbin/nologin]=2

Bash[/bin/sync]=1

/bin/bash:1

/sbin/nologin:2

/bin/sync:1

8.5 case

Syntax: switch (expression) {case VALUE or/regexp/: statement1, Statement2,... default:statement1, ...}

8.6 Break and continue

Often used in loops or case statements

8.7 Next

End the processing of the bank's text in advance and proceed to the next line; For example, the following command displays users whose ID numbers are odd:

# awk-f: ' {if ($3%2==0) next;print $1,$3} '/etc/passwd

Using arrays in nine awk

9.1 Arrays

Array[index-expression]

Index-expression can use arbitrary strings; It is important to note that if a data group element does not exist beforehand, awk automatically creates this element and initializes it to an empty string when it is referenced, so you need to use the index in to determine whether an element exists in a data group. Array in the same way.

To enumerate through each element of a group, you need to use a special structure like this:

for (var in array) {statement1, ...}

Where Var is used to reference array subscripts, not element values;

Example:

Netstat-ant | awk '/^tcp/{++s[$NF]} END {for (a in S) print A, s[a]} '

Each occurrence of a row that is matched by the/^tcp/pattern, the array s[$NF] adds 1,nf to the last field of the currently matched row, where its value is used as an element index of the array s;

awk ' {counts[$1]++}; END {for (URL in counts) print Counts[url], url} '/var/log/httpd/access_log

Used in the same way as the previous example, to count the amount of IP access in a log file

9.2 Deleting an array variable

Deleting an array index from a relational array requires the use of the Delete command. Use the format as:

Delete Array[index]

X. Built-in functions for awk

Split (string, array [, Fieldsep [, SEPs]])

Function: Separates string representations of strings with fieldsep as delimiters, and saves the separated results to an array with an array name, which is labeled as a sequence starting from 1;

# Netstat-ant | awk '/:80\>/{split ($5,clients, ":"); Ip[clients[1]]++}end{for (i in IP) {print ip[i],i}} ' | Sort-rn | Head-50

# Netstat-tan | awk '/:80\>/{split ($5,clients, ":"); Ip[clients[4]]++}end{for (A in IP) print Ip[a],a} ' | Sort-rn | Head-50

# DF-LH | awk '!/^file/{split ($5,percent, "%"); if (percent[1]>=20) {print}} '

Length ([string])

Function: Returns the number of characters in string strings;

SUBSTR (String, start [, length])

Function: Take a substring in string, start with start, take length, start counting from 1;

System (Command)

Function: Execute system command and return the result to the awk command

Systime ()

Function: Take system current time

ToLower (s)

Function: Convert all letters in s to lowercase

ToUpper (s)

Function: Convert all letters in s to uppercase

XI. user-defined functions

The custom function uses the function keyword. The format is as follows:

function f_name ([variable])

{

Statements

}

The function can also return a value using the return statement in the form "return value".

Netstat-tn |awk '/^tcp/{state[$NF]++}end{for (s in state) {printf "%-15s 5d\n", S,state[s]} '

Count the number of connections per connection

Practice:

1, statistics on the current system of each client IP connection is in the Time_wait connection state number;

2, the Statistics PS aux command execution, the current system on the status of the number of processes;

[[Email protected] ~]# PS Aux|awk '!/^user/{state[$8]++}end{for (i in state) {printf "%-15s%d\n", I,state[i]}} '

s< 2

S<SL 1

Ss 18

STAT 1

SN 1

S 94

ss+ 7

SSL 4

r+ 1

S+ 1

Sl 6

S<s 1

3, the Statistics PS aux command execution, the current system on the number of user processes;

[[Email protected] ~]# PS aux |awk '!/^user/{state[$1]++}end{for (i in state) {printf "%s%d\n", I,state[i]}} '

RPC 1

Dbus 1

Named 1

68 2

Daemon 4

Postfix 2

Rpcuser 1

Root 122

4, display PS aux command execution, the current system on its vsz (virtual memory set) is greater than 10000 of the process and its PID;

PS Aux|awk '!/^user/{if ($5<10000) next;print $2,$5, $NF} '

PS Aux|awk '!/^user/{if ($5>10000) print $2,$5, $NF} '

Awk II, AWK advanced