Linux Text processing the Three Musketeers of awk detailed

Objective

Awk is a powerful report generator, unlike SED and grep, which focuses on how to better display textual information, commonly used with statistics and formatted output. awk acts as a miniature shell with its own set of grammatical structures, such as looping structures, arrays, conditional judgments, functions, built-in variables, and so on. Handles objects generally plain text files or plain text information.

Usage explanation

Basic syntax

awk [Options] ' program ' File File ... awk [options] ' pattern{action} ' file File ...-f CHAR: Input delimiter

The output of awk

Print Item1, item2,...

① each item is separated by commas, and the output is delimited with an output delimiter

Each item of the ② output can be a string or numeric value, a field of the current record, a variable, or an expression of awk, and the value will be implicitly converted to a string after the output

③print after item if omitted, equivalent to print $, if the output is blank, use Pirnt ""

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/16/wKiom1U_UF6C225dAADyOytP9sg195.jpg "title=" 1.jpg " alt= "Wkiom1u_uf6c225daadyoytp9sg195.jpg"/>

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/12/wKioL1U_UiHiji1HAAAvfNIw4nU492.jpg "title=" 2.jpg " alt= "Wkiol1u_uihiji1haaavfniw4nu492.jpg"/>

Mode

①REGEXP: Format is/pattern/, only rows that are matched to/pattern/are processed

②expression: An expression that satisfies a condition when the result is a non-0 or non-empty string, processing only rows that satisfy the condition

③ranges: Line range, previous address delimitation, startline, endline, processing only rows within the range

④begin/end: Special mode that executes only once before the program of the awk command runs (BEGIN) or after run (END)

⑤empty: Empty mode, matches any row

The variables of awk

Built-in variables

Fs:field seperator, input field delimiter Rs:record seperator, output line delimiter ofs:output field seperator, output field delimiter; Ors:outpput row seperator , the line delimiter at the output;

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/19/wKiom1U_g7HxfP9xAAEhkRr3gbc649.jpg "title=" 3.jpg " alt= "Wkiom1u_g7hxfp9xaaehkrr3gbc649.jpg"/>

Nr:numbers of Record, number of rows; count all files together

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/15/wKioL1U_hUyDIjBkAAISe6e2Rp8586.jpg "title=" 4.jpg " alt= "Wkiol1u_huydijbkaaise6e2rp8586.jpg"/>

Nf:numbers of field, number of fields

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/1A/wKiom1U_hOuwxJBtAAILKk_cxQ0764.jpg "title=" 5.jpg " alt= "Wkiom1u_houwxjbtaailkk_cxq0764.jpg"/>

FNR: number of rows; Each file is counted separately;

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/16/wKioL1U_h2aipO-nAAII0RfwZxQ302.jpg "title=" 6.jpg " alt= "Wkiol1u_h2aipo-naaii0rfwzxq302.jpg"/>

ARGV: Array, save command itself this character, awk ' {print $} ' 1.txt 2.txt, meaning argv[0] Save AWKARGC: Save the number of arguments in the awk command Filename:awk the name of the current file being processed

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/16/wKioL1U_iTKB85BIAADEDhXwBGI806.jpg "title=" 7.jpg " alt= "Wkiol1u_itkb85biaadedhxwbgi806.jpg"/>

Variable can be customized

-V Var_name=value #变量名区分字符大小写

① can define variables in program

② can customize variables in the command line via the-v option

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1B/wKiom1U_itSDYtJ9AABZSxKbw7M357.jpg "title=" 9.jpg " alt= "Wkiom1u_itsdytj9aabzsxkbw7m357.jpg"/>

Awk's printf command

Command use format: printf format, item1, item2,...

① to specify the format

The ② does not wrap, and you need to give \ n if you want to wrap the line

The ③format is used to specify its output format for each subsequent item

The format indicator is preceded by a character:%c: The ASCII code that displays the character,%d,%i: decimal integer,%e,%e: The scientific notation displays the value,%f: Displays the floating-point number,%g,%g: Displays the value in scientific notation format or floating-point format;%s: display string; %u: Displays unsigned integers; Percent: shows% itself; modifier: #: Display width-: Left align +: Displays the symbol for the value. #: Value Accuracy

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/17/wKioL1U_klzy_4E9AADKmR_vpkA142.jpg "title=" 10.jpg "alt=" Wkiol1u_klzy_4e9aadkmr_vpka142.jpg "/>

awk output Redirection

Print Items > Output-file #保存到某文件print items >> output-file #追加到某文件print Items | Command #使用管道交给某些命令处理特殊文件描述符:/dev/stdin: Standard input/dev/stdout: standard output/dev/stderr: Error Output

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/17/wKioL1U_lxfALFa0AABmtinGWro867.jpg "title=" 11.jpg "alt=" Wkiol1u_lxfalfa0aabmtingwro867.jpg "/>

Awk's operator

Arithmetic operators: X+yx-yx*yx/yx**y, X^yx%y-x: Negative +x: Convert to numeric string operator: Join assignment operator: =+=-=*=/=%=^=**= ++--If the pattern itself is the = sign, write as/=/comparison operator: <&L t;=>>===!=~: pattern matching, the left string can be matched to the right pattern to true, otherwise false;!~: Mismatch is true, matching is false logical operator: &&: With | | : or conditional expression: selector?if-true-expression:if-false-expression function call: Function_name (ARGU1,ARGU2)

Example: Determine if the UID is greater than or equal to 500, if true is displayed as a normal user, if False is displayed as system or administrative user

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/1B/wKiom1U_l2jxNGekAAFXmSMCCvY816.jpg "title=" 12.jpg "alt=" Wkiom1u_l2jxngekaafxmsmccvy816.jpg "/>

Common action

①expressions: expression Assignment, etc.

②control statements: Conditional statement, control statement if while ...

③compound statements: Combined statement

④input statements: Input statement

⑤output statements: Output statement

Control statements

If-else

Format: if (condition) {then body} else {else body}

Example: uid>=500 users are displayed as normal users, otherwise they are displayed as administrative users

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/17/wKioL1U_mneBbwgJAAFuEbGQPlk709.jpg "title=" 13.jpg "alt=" Wkiol1u_mnebbwgjaafuebgqplk709.jpg "/>

Example: Displaying a row in a file with a number of fields greater than or equal to 8

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/17/wKioL1U_nBmBybE_AAGX1Y4j-3o403.jpg "title=" 14.jpg "alt=" Wkiol1u_nbmbybe_aagx1y4j-3o403.jpg "/>

While

Format: while (condition) {while body}

Example: Show only odd rows

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1B/wKiom1U_m3HCW_3hAAGunkdiHr4780.jpg "title=" 15.jpg "alt=" Wkiom1u_m3hcw_3haagunkdihr4780.jpg "/>

Example: Output only fields that are greater than 6 per row

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/17/wKioL1U_ngrhErNFAACwP2SDpXk687.jpg "title=" 16.jpg "alt=" Wkiol1u_ngrhernfaacwp2sdpxk687.jpg "/>

Do-while Cycle

Format: do {Do-while body} while (condition)

Example: Output 1 to 100 of the sum

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/1B/wKiom1U_nh-i4KQDAABFr2K9FPA256.jpg "title=" 17.jpg "alt=" Wkiom1u_nh-i4kqdaabfr2k9fpa256.jpg "/>

For loop

Format: for (variable assignment; condition, iteration process) {for body}

Example: displaying odd lines

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/17/wKioL1U_obKDIK_wAAGuKDKTozk701.jpg "title=" 18.jpg "alt=" Wkiol1u_obkdik_waagukdktozk701.jpg "/>

Example: output a field greater than 6 per line

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1B/wKiom1U_oLGTFpyoAACtQZhmz7I661.jpg "title=" 19.jpg "alt=" Wkiom1u_olgtfpyoaactqzhmz7i661.jpg "/>

A For loop can be used to iterate over an array element

Syntax: for (i in array) {for body}

Example: Save array in a, save array subscript in B

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/1B/wKiom1U_o26DyjOKAABO_L2KV80236.jpg "title=" 20.jpg "alt=" Wkiom1u_o26dyjokaabo_l2kv80236.jpg "/>

Case statement

Syntax: switch (expression) {case VALUE or/rgeexp/: statement1; ... default:stementn}

Loop control

Break #退出当前循环continue #提前结束本轮循环, go straight to the lower loop

Next

End the processing of the bank in advance and proceed to the next line

Example: Show users with an odd user ID

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/18/wKioL1VAL6iw5igUAACrCT5GwaY852.jpg "title=" 21.jpg "alt=" Wkiol1val6iw5iguaacrct5gway852.jpg "/>

Example: A user who outputs odd rows

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/1C/wKiom1VALsLSwdz8AADDihktHA0063.jpg "title=" 22.jpg "alt=" Wkiom1valslswdz8aaddihktha0063.jpg "/>

Array

Associative arrays

Array[index-expression]index-expression: Any string can be used; If an array element does not exist beforehand, awk automatically creates the element and initializes it to an empty string when referenced, so that the "index in array" format must be used to determine if an element exists

To enumerate through each element of the group, you need to use a special structure like this

For (var. array) {for body} #其var会遍历array的索引

Example: count the number of times each network connection appears

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/18/wKioL1VANB7iriwnAABjVRdXEvI414.jpg "title=" 23.jpg "alt=" Wkiol1vanb7iriwnaabjvrdxevi414.jpg "/>

Example: Statistics of IP Access in Web service access logs

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/18/wKioL1VANaGQQlKKAAB2dDiTFMY916.jpg "title=" 24.jpg "alt=" Wkiol1vanagqqlkkaab2dditfmy916.jpg "/>

To delete an array element

Deleting an array index in a relational array requires the use of the Delete command

Format: Delete Array[index]

Awk's built-in functions

Split (String,array[,fieldsep[,seps]]) function: The string represented by Fieldsep as a delimiter to slice, and the result of the slice is saved to an array of the names of arrays; array subscript starting from 1 This function has a return value, The number of elements after which the return value is sliced

Example: Count the IP of a connection request

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1D/wKiom1VAN_GASkFRAABsAW8UHyI744.jpg "title=" 25.jpg "alt=" Wkiom1van_gaskfraabsaw8uhyi744.jpg "/>

Length (String) function: Returns the lengths of a given string substr (String,start[,length]) function: Takes a substring from a string, from start to the starting position for the length of the substring system (command) Function: Executes the system command and returns the result to the awk command Systime () function: The Systime function returns the number of whole seconds from January 1, 1970 to the current time (excluding leap years) ToLower (s) Function: Convert all letters in s to lowercase toupper (s) function: Capitalize all letters in s

The end

Well, the use of awk comes first here, awk is not only strong, but also look at my head, the interest can be in-depth study, this article will not go further. The above is only for individual learning to organize, if there are mistakes, big God do not spray ~ ~ ~

This article is from the "North Scholar" blog, please make sure to keep this source http://scholar.blog.51cto.com/9985645/1640179

Linux Text processing the Three Musketeers of awk detailed