Awk Usage Details

Awk usage description awk: The Chinese meaning is that the Report Generator can format and display the information based on the information we enter, and the defined information is more beautiful (intuitive) the new awk (nawk) is implemented on windows, and gawk and awk are implemented on linux. Awk is a programming language used to process text and data in linux/unix. Data can come from standard input, one or more files, or the output of other commands. It supports advanced functions such as user-defined functions and dynamic regular expressions, and is a powerful programming tool in linux/unix. It is used in the command line, but more is used as a script. Awk processes text and data in this way. It scans files row by row, from the first row to the last row, and looks for rows matching specific patterns, and perform the operations you want on these rows. If no processing action is specified, the matched rows are displayed to the standard output (screen). If no mode is specified, all the rows specified by the operation are processed. Awk represents the first letter of the last name of the author, namely Alfred Aho, Brian Kernighan, and Peter Weinberger. Gawk is the GNU version of awk. It provides Bell Labs and GNU extensions. The following describes awk. The basic format used by the awk is 123 # awk [options] 'script' file1 file2 ,... # awk [options] 'pattern' {action} 'file1 file2 ,... the Common commands used in the preceding combination are refined: print (output), and printf (print) awk read a line of text from the file at a time according to the defined mode, awk slices the extracted text and splits each line according to the delimiter. If we have a line of text: this is a test. it uses blank characters as separators to separate them and cut them into four slices: this is a test. variables can be used for the four slices, corresponding to $1, $2, $3, and $4, respectively. The following is a demonstration, see how to implement the powerful functions of awk I. awk output: print 12print format: print item1, item2 ,... output highlights: 1 And each item is separated by commas, while the output is separated by blank characters. 2. The output item can be a string or value, and the field of the current record (for example, $1) variable or awk expression. The value is first converted to a string and then output. 3. The item after the print command can be omitted. In this case, the function is equivalent to print $0. Therefore, if you want to output blank lines, use print ""; instance analysis: to display the entire text (note: this is only a test, so there is only one line) # awk '{print $ N}' test.txt this is a test. # awk '{print $0}' test.txt this is a test. the first parameter # awk '{print $1} 'test.txt this shows the second parameter in the text line # awk' {print $2} 'test.txt is show the text # awk '{print $4}' test.txt tes T. parameters for the first and second lines of the displayed text # awk '{print $1, $2}' test.txt this is the first and second parameters in the line of the displayed text, you can add the # awk 'BEGIN {OFS = "#"} {print $1, $2}' test.txt this # is to display parameters in the text, you can also add the modifier # awk 'in in {OFS = ":"} {print $1, $2, $3, $4} 'test.txt this: is: a: test. display parameters in the text. You can also add modifiers and strings # awk 'in in {OFS = ":"} {print $1, "hello ", $2} 'test.txt this: hello: is output three rows, note that \ n # awk 'BEGIN {print "line one \ nline two \ nline three"}' line oneline twoline thr must be added for line feed. Ee 2. awk variable 2.1 awk built-in variable Record variable: 1234FS: field separator, field separator RS: Record separator Used for reading text information OFS: output Filed Separator: Field Separator Used for Output text ORS: Output Row Separator: The Row Separator Used for Output text 2.2 awk built-in variable data variable: NR: The number of input records, number of records processed by the awk command; If there are multiple files, this Number will count the rows of the processed files in a unified manner; NF: Number of Field, Number of fields of the current record; FNR: different from NR, FNR is used to record the row being processed as the total number of rows processed in the current file. instance analysis: number of lines in the displayed text # awk '{print NR}' test.txt 1 shows a row with a total of several parameters # Awk '{print NF}' test.txt 4 is the last parameter for text display # awk '{print $ NF}' test.txt test. view multiple texts at the same time # awk '{print FNR}' test.txt/etc/fstab1 shows the number of rows of the first text 1 shows the first number of rows of the second text, 1. Push 2 shows the second row of the second text. 3 shows the third row of the second text. ARGV: array. Save the string of the command line, for example, in the Command awk '{print $0}' a.txt B .txt, argv1_01_awk and argv1_11_a.txt; ARGC: number of parameters of the awk command; FILENAME: name of the file processed by the awk command; ENVIRON: the associated array of the Current shell environment variable and its value; 1234 view the environment variable # awk 'in in {print ENVIRON ["PATH"]} '/usr/curb Eros/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin: /usr/bin:/usr/local/apache/bin:/usr/local/mysql/bin: /root/bin2.3 the custom variable gawk allows you to customize your own variables for use in program code. The naming rules for variable names are the same as those for most programming languages, only letters, numbers, and underscores are allowed, and cannot begin with a number. Gawk variable names are case sensitive. 2.3.1 assign values to variables in the script. assign values to variables in gawk using the assign value statement. 2.3.2 use the assign value variable in the command line. The gawk command can also assign values to variables outside the script, and reference it in the script. Instance analysis: 12345 outputs a variable and assigns a value # awk 'in in {var = "variable testing "; print var} 'variable testing # awk-v var = "variable testing" 'In in {print var} 'variable testing III. awk printing tool: the format of the printf 12printf command: printf format, item1, item2 ,... printf highlights: 1. The biggest difference between the printf and the print command is that printf needs to specify the format; 2. format is used to specify the output format of each item; 3. the printf statement does not automatically print line breaks. You need to add the \ n format specifiers starting with %, followed by a character, as follows: % c: display the first character % d, % I: decimal INTEGER OF THE FIRST parameter; % e, % E: Scientific notation value; % f: explicit Floating Point Number; % g, % G: Numeric value displayed in scientific notation or floating point format; % s: Display string; % u: unsigned integer; %: display % itself; modifier: N: display width;-: Left alignment; +: Display numeric symbol; example: # awk '{printf "% c \ n ", $1} 'test.txt t displays the first letter of the first parameter # awk' {printf "% s \ n ", $1} 'test.txt this shows the first parameter # awk '{printf "% 10s \ n", $1}' test.txt this shows 10 strings, the default value is the right alignment # awk '{printf "%-10s \ n", $1}' test.txt this indicates the left alignment, there are 6 empty strings # awk '{printf "%-10 s, %-10s \ n", $1, $2}' test.txt this, is # awk '{printf "%-10 s %-10s \ n ", $1, $2} 'test.txt this is # awk-F: '{printf" % 5d ", $3} '/etc/passwd 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 99 81 28 69 77 38 70 32 47 51 74 16 68 100 43 42 500 501 502 5032002 2003 2004 4004 2033 2034 4005 4006 4007 4011 4017 4018 4019 4026 4027 4028 48 25 101 # awk-F: '{printf "%-5d \ n", $3}'/etc/passwd left aligned # awk-F: '{printf "% + 5d \ n ", $3} '/etc/passwd right aligned # awk-F:' {printf "%-15 s % I \ n", $ 1, $3} 'test.txt this is a test. 0 IV. output redirection print items> output-fileprint items | command special file descriptor:/dev/stdin: Standard Input/dev/sdtout: standard output/dev/stderr: Error output/dev/fd/N: a specific file descriptor. For example,/dev/stdin is equivalent to/dev/fd/0. Example: # awk-F: '{printf "%-15 s % I \ n", $1, $3> "/dev/stderr"} '/etc/passwdroot 0bin 1 daemon 2adm 3lp 4 sync 5 shutdown 6 halt 7 5. awk operators: 5.1 Arithmetic Operators:-x: negative value + x: Convert to numeric value; x ^ y: x ** y: x * y: Multiplication x/y: Division x + y: x-y: x % y: 5.2 string OPERATOR: there is only one operator, which is used to connect strings. 5.3 value assignment operator: =, + =,-=, * =,/=, % =, ^ =, ** =, ++, -- Note that if a mode is =, when/=/is used, a syntax error may occur. Replace it with/[=]/. In the 5.4 Boolean awk, any non-0 or non-null string is true, otherwise, it is false. 5.5 comparison operator: x <y x less than y is true; otherwise, false x <= y x less than or equal to y is true, otherwise, false x> y x is greater than y. Otherwise, false x> = y x is greater than or equal to y. Otherwise, false x = y x is always equal to y, otherwise false x! = Y x is not equal to y, which is true, and vice versa ~ Y True if the string x matches the regexp denoted by y. x !~ Y True if the string x does not match the regexp denoted by y. subscript in array True if the array has an element with thesubscript subscript.5.5 logical relational character between Expressions: &: logical and |: logical or 5.6 conditional expression: equivalent to if statement example: if a is greater than B, a is max, and B is maxa = 3b = 4a> B? A is max: B is max5.7 function call: function_name (para1, para2) 6. awk mode: awk 'program 'input-file1 input-file2... the program is: pattern {action }... 6.1 common pattern types: 1. Regexp: regular expression. Format:/regular expression/For example: Display rows starting with r # awk-F: '/^ r/{print $1}'/etc/passwdrootrpcrpcuser2, expresssion: expression. Conditions are met when the value is not 0 or non-null, for example, $1 ~ /Foo/or $1 = "magedu", using the operator ~ (Matching) and ~! (Mismatch ). Example: show the user and ID number whose ID number is less than or equal to 2 # awk-F: '$3-1 <5 {print $1, $3} '/etc/passwdroot 0bin 1 daemon 2 # awk-F:' $3 <= 5 {print $1, $3} '/etc/passwdroot 0bin 1 daemon 2. Find the user whose default shell is bash # awk-F:' $7 ~ "Bash $" {print $1, $7} '/etc/passwdroot/bin/bashstudent/bin/bashvisitor/bin/bashmyuseradd/bin/bash: the default shell is not a bash user # awk-F:' $7! ~ "Bash" {print $1, $7} '/etc/passwdbin/sbin/nologindaemon/sbin/nologinadm/sbin/nologinlp/sbin/nologin3, Ranges: specified matching range, the format is pat1. If pat2 shows a user whose ID number is 0 or whose shell is/sbin/nologin, # awk-F: '$3 = 0, $7 ~ "Nologin" {print $1, $3, $7} '/etc/passwdroot 0/bin/bashbin 1/sbin/nologin4, BEGIN/END: special mode, run only once before or before the awk command is executed # awk-F: '$3 = 0, $7 ~ "Nologin" {printf "%-10 s %-20s \ n", $1, $7} '/etc/passwdroot/bin/bash bin/sbin/nologin can display the header # awk-F: 'In in {print "Username shell"} {printf "%-10 s %-20s \ n", $1, $7} '/etc/passwdUsername shellroot/bin/bash bin/sbin/nologin daemon/sbin/nologin adm/sbin/nologin display header and table tail # awk-F: 'In in {print "Username shell"} {printf "%-10 s %-20s \ n", $1, $7} END {print "end of report"} '/etc/passwdUsername shellroot/bin/B Ash bin/sbin/nologin daemon/sbin/nologin adm/sbin/nologin end of report5, Empty (null mode): match any input line, match each row in the file # awk-F: '{printf "%-10 s %-20s \ n", $1, $7} '/etc/passwdroot/bin/bash bin/sbin/nologin daemon/sbin/nologin adm/sbin/nologin6.2 common actions (Actions) 1. Expressions: expression 2. Control statements: Control statement (if, while... do, .... wait) 3. Compound statements: Composite Statement 4. Input statements: Input statement 5. Output statements: Output statement/regular expression/: Use The Extension Set of the identifier. Relational Expression: You can use the Relational operators in the following operator table to perform operations. It can be a string or number comparison, for example, $2> % 1. Select a row whose second field is longer than the first field. Pattern Matching expression: pattern, pattern: Specifies the range of a row. This syntax cannot include the BEGIN and END modes. BEGIN: Specifies the action that occurs before the first input record is processed. You can set global variables here. END: The action that occurs after the last input record is read. VII. Control statement: 7.1 if-else Syntax: if (condition) {then-body} else {[else-body]} instance analysis: if the user is root, "admin" is displayed; otherwise, "Common user" # awk-F: '{if ($1 = "root") print $1, "admin"; else print $1, "Common user"} '/etc/passwdroot adminbin Common userdaemon Common useradm Common userlp Common user this is formatted in the preceding syntax, looks more beautiful # awk-F: '{if ($1 = "root") printf "%-15 s: % s \ n", $1, "Admin "; else printf "%-15 s: % s \ n", $1, "Common User "}' /Etc/passwdroot: Adminbin: Common Userdaemon: Common Useradm: Common Userlp: Common User: count the number of users whose ID number is greater than or equal to 500 # awk-F: -v sum = 0' {if ($3> = 500) sum ++} END {print sum} '/etc/passwd217.2 while Syntax: while (condition) {statement1; statment2 ;...} instance analysis: displays strings greater than or equal to 8 (each string must be judged) # awk-F: '{I = 1; while (I <= NF) {if (length ($ I) >=8) {print $ I }; I ++} '/etc/passwd/bin/bash/sbin/nologin/var/adm/sbi N/nologin/var/spool/lpd: # awk-F: '{I = 1; while (I <= NF) {if (length ($ I) >=4) {print $ I }; I ++} '/etc/passwdrootroot/root/bin/bash/bin/sbin/nologindaemondaemon7.3 do-while Syntax: do {statement1, statement2 ,...} while (condition) indicates that the string is less than or equal to 4 (note that when the do statement is used, the condition is first executed) # awk-F: '{I = 1; do {print $ I; I ++} while (I <= 3)} '/etc/passwdrootx0bin7.4 for syntax: for (variable assignment; condition; iteration process ){ Statement1, statement2 ,...} example: 1. Show # awk-F: '{for (I = 1; I <= NF; I ++) with a string greater than or equal to 4) {if (length ($ I)> = 4) {print $ I }}'/etc/passwdrootroot/root/bin/bash2: # awk-F: '{for (I = 1; I <= 3; I ++) print $ I} '/etc/passwdrootx0binfor loop can also be used to traverse array elements: Syntax: for (I in array) {statement1, statement2 for example: display # awk-F: '$ NF! ~ /^ $/{BASH [$ NF] ++} END {for (A in BASH) {printf "%-15 s: % I \ n",, BASH [A]} '/etc/passwd/bin/sync: 1/bin/bash: 20/sbin/nologin: 31/sbin/halt: 1/etc/tcsh: 1/sbin/shutdown: 17.5 case Syntax: switch (expression) {case VALUE or/REGEXP/: statement1, statement2 ,... default: statement1 ,...} 7.6 break and continue are often used in loop or case statements. 7.7 next ends processing the text of this line in advance and then processes the next line. For example, the following command displays users with an odd ID: # awk-F: '{if ($ 3% 2 = 0) next; print $1, $3} '/etc/passwdbin 1adm 3 sync 5 halt 7 news 9 operator 11 8. Use an array 8.1 array [index-expression] index-expression to use any string in awk; note that if a data group element does not exist in advance, awk will automatically create the element and initialize it as an empty string at the time of reference. Therefore, to determine whether an element exists in a data group, use the index in array method. To traverse every element in the array, use the following special structure: for (var in array) {statement1 ,...} show the number of various shells and # awk-F: '{shell [$ NF] ++} END {for (A in shell) {print, shell [A]} '/etc/passwd 1/bin/sync 1/bin/bash 20/sbin/nologin 31/sbin/halt 1/etc/tcsh 1/sbin/ shutdown 1 where, var is used to reference the array subscript instead of the element value. It counts the number of tcp connection States ($6 is also the last field, and $ NF can be used) # netstat-ant | awk '/^ tcp/{STATE [$ NF] ++} END {for (a in STATE) print, STATE [a]} 'Listen 11 ESTABLISHED 2 appears every time /^ For the row matching the tcp/mode, add 1 to the array S [$ NF], and NF is the last field of the currently matched row, here, the value is used as the element index of array S. The IP address and number of local accesses are counted # awk '{count [$1] ++} END {for (ip in count) {printf "%-20 s: % d \ n", ip, count [ip]} '/var/log/httpd/access_log192.168.10.1: 1172.16.50.5: 10 its usage is the same as that in the previous example. It is used to count the access volume of an IP address in a log file. 8.2 The delete command is required to delete an array variable from a relational array to delete an array index. Use the built-in function split (string, array [, fieldsep [, seps]) in the format of delete array [index] 9 and awk: the string is separated by fieldsep, and the results are saved to an array named after array. The subscript of the array is a sequence starting from 0; # netstat-ant | awk '/: 80 \>/{split ($5, clients ,":"); IP [clients [1] ++} END {for (I in IP) {print IP [I], i} '| sort-rn | head-50 length ([string]) function: return the number of characters in the string; substr (string, start [, length]) function: take the substring in the string, start from start, and get the length; start starts from 1 to count; s Ystem (command) function: Execute the system command and return the result to the awk command Iime () function: Get the current system time tolower (s) function: convert all letters in s to lowercase toupper (s). function: Convert all letters in s to uppercase 10. Use the function keyword for user-defined function-defined functions. The format is as follows: function F_NAME ([variable]) {statements} can also return values using the return statement. The format is "return value ".