Linux awk command details, awk command details

Source: Internet
Author: User

Linux awk command details, awk command details


Awk is a powerful text analysis tool. Compared with grep search and sed editing, awk is particularly powerful in data analysis and report generation. To put it simply, awk refers to reading files row by row. Each line is sliced with spaces as the default separator, and the cut part is analyzed and processed.

Awk has three different versions: awk, nawk, and gawk, which are generally gawk and gawk is the GNU version of AWK.

Awk is named from the first letter of its founder Alfred Aho, Peter Weinberger, and Brian Kernighan. In fact, AWK does have its own language: AWK programming language. The three creators have formally defined it as "style scanning and processing language ". It allows you to create short programs that read input files, Sort data, process data, perform calculations on input, and generate reports. There are countless other functions.



Awk '{pattern + action}' {filenames}

Although the operation may be complex, the syntax is always like this. pattern indicates the content that AWK searches for in the data, and action is a series of commands executed when matching content is found. Curly braces ({}) do not always appear in the program, but they are used to group A series of commands according to a specific mode. Pattern is the regular expression to be expressed and enclosed by a slash.

The most basic function of the awk language is to browse and extract information based on specified rules in a file or string. Only after awk extracts information can other text operations be performed. A complete awk script is usually used to format information in a text file.

In general, awk is a row of files for processing. Every time an awk receives a line of files, it then executes the corresponding command to process the text.



There are three methods to call awk


1. Command Line Method awk [-F field-separator] 'commands' input-file (s)

Commands is a real awk command, and the [-F domain separator] is optional. Input-file (s) is a file to be processed.

In awk, each line of a file is called a domain separated by a domain separator. Generally, the default domain separator is a space without specifying the-F domain separator. 2. shell script

Insert all the awk commands into a file and make the awk program executable. Then, the awk command interpreter serves as the first line of the script and is called by typing the script name again.

Equivalent to the first line of shell script :#! /Bin/sh can be changed :#! /Bin/awk 3. Insert all the awk commands into a separate file, and then call: awk-f awk-script-file input-file (s)

Among them, the-f option loads the awk script in the awk-script-file, and the input-file (s) is the same as above.


This chapter focuses on the command line method.


Entry instance

Assume that the output of last-n 5 is as follows:

[Root @ www ~] # Last-n 5 <= retrieve only the first five rows

Root pts/1 Tue Feb 10 still logged in root pts/1 Tue Feb 10)

Root pts/1 Mon Feb 9)

Dmtsai pts/1 Mon Feb 9)

Root tty1 Fri Sep 5)

If only the five most recently logged on accounts are displayed

# Last-n 5 | awk '{print $1 }'






The awk workflow is as follows: Read a record with '\ n' line breaks, divide the record into fields according to the specified domain separator, and fill in the fields. $0 indicates all fields, $1 indicates the first domain, and $ n indicates the nth domain. The default domain separator is "Blank key" or "[tab] Key", so $1 indicates the logon user, $3 indicates the logon user ip, and so on.


If you only display the/etc/passwd account

# Cat/etc/passwd | awk-F': ''{print $1} 'root




This is an example of awk + action. action {print $1} is executed on each line }.

-F specifies that the domain separator is ':'.


If only the/etc/passwd account and shell corresponding to the account are displayed, the account and shell are separated by the tab key.

# Cat/etc/passwd | awk-F': ''{print $1" \ t "$7} 'root/bin/bash

Daemon/bin/sh sys/bin/sh


If only the shell corresponding to the/etc/passwd account and account is displayed, the account and shell are separated by commas, and the name and shell column are added to all rows, add "blue,/bin/nosh" to the last line ".


Cat/etc/passwd | awk-F': ''BEGIN {print" name, shell "} {print $1", "$7} END {print" blue, /bin/nosh "} 'name, shell


Daemon,/bin/sh bin,/bin/sh sys,/bin/sh ....



The awk workflow is as follows: first execute BEGING, then read the file, read a record with/n line breaks, then divide the record into Domains Based on the specified domain separator, and fill in the domain, $0 indicates all domains, $1 indicates the first domain, $ n indicates the nth domain, and then starts the action corresponding to the execution mode. Then read the second record until all the records are read and the END operation is executed.


Search for all rows with the root keyword in/etc/passwd.

# Awk-F: '/root/'/etc/passwd root: x: 0: 0: root:/bin/bash

This is an example of pattern. Only the row matching pattern (root here) can execute action (no action is specified, and the content of each row is output by default ).

Regular Expressions are supported in search, for example, awk-F: '/^ root/'/etc/passwd.


Search for all rows with the root keyword in/etc/passwd and display the corresponding shell

# Awk-F: '/root/{print $7}'/etc/passwd/bin/bash

Action {print $7} is specified here}


AwkBuilt-in Variables

Awk has many built-in variables used to set environment information. These variables can be changed. The following lists the most common variables.


Number of ARGC command line parameters

ARGV command line parameter arrangement

ENVIRON supports the use of system environment variables in the queue

FILENAME awk browsed file name

Number of FNR browsing file records

FS sets the input domain separator, which is equivalent to the command line-F Option

Number of NF browsing records

Number of records read by NR

OFS output domain Separator

ORS output record Separator

RS control record delimiter


In addition, the $0 variable refers to the entire record. $1 indicates the first domain of the current row, $2 indicates the second domain of the current row, and so on.


Statistics/etc/passwd: file name, row number of each row, column number of each row, corresponding to the complete row content:

# Awk-F': ''{print" filename: "FILENAME", linenumber: "NR", columns: "NF", linecontent: "$0} '/etc/passwd filename:/etc/passwd, linenumber: 1, columns: 7, linecontent: root: x: 0: 0: root:/root: /bin/bash

Filename:/etc/passwd, linenumber: 2, columns: 7, linecontent: daemon: x: 1: 1: daemon:/usr/sbin:/bin/sh filename: /etc/passwd, linenumber: 3, columns: 7, linecontent: bin: x: 2: bin:/bin/sh filename:/etc/passwd, linenumber: 4, columns: 7, linecontent: sys: x: 3: 3: sys:/dev:/bin/sh


Use printf instead of print to make the code more concise and easy to read

Awk-F': ''{printf (" filename: % 10 s, linenumber: % s, columns: % s, linecontent: % s \ n ", FILENAME, NR, NF, $0)} '/etc/passwd



Both print and printf are provided in awk.

The print function can be a variable, a value, or a string. The string must be referenced in double quotation marks and the parameters must be separated by commas. If there are no commas (,), the parameters are connected together and cannot be distinguished. Here, the comma serves the same purpose as the separator of the output file, except that the latter is a space.

The printf function is similar to the printf function in C language. It can format strings. When the output is complex, printf is easier to use and the code is easier to understand.



 Variables and assignments

In addition to the built-in variables of awk, awk can also customize variables.

The following table lists the number of accounts in/etc/passwd.

Awk '{count ++; print $0;} END {print "user count is", count}'/etc/passwd root: x: 0: 0: root: /root:/bin/bash


User count is 40

Count is a custom variable. In the previous action {}, only one print exists. In fact, print is only a statement, and action {} can have multiple statements separated by a comma.


The count is not initialized here. Although the default value is 0, it is recommended to initialize it as 0:

Awk 'in in {count = 0; print "[start] user count is", count} {count = count + 1; print $0 ;} END {print "[end] user count is", count} '/etc/passwd [start] user count is 0 root: x: 0: 0: root:/root: /bin/bash


[End] user count is 40


Count the number of bytes occupied by files in a folder

Ls-l | awk 'in in {size = 0 ;}{ size = size + $5 ;}end {print "[END] size is", size }'

[End] size is 8657198


If the unit is M:

Ls-l | awk 'in in {size = 0 ;}{ size = size + $5 ;}end {print "[END] size is", size/1024/1024, "M"} '[end] size is 8.25889 M

Note: statistics do not include subdirectories of folders.


Condition Statement

The condition statements in the awk are used for reference in the C language. See the following declaration method:


If (expression ){




} If (expression ){


} Else {


} If (expression ){


} Else if (expression1 ){


} Else {





Count the number of bytes occupied by files in a folder and filter out files of 4096 size (usually folders ):

Ls-l | awk 'in in {size = 0; print "[start] size is", size} {if ($5! = 4096) {size = size + $5 ;}end {print "[END] size is", size/1024/1024, "M"} '[end] size is 8.22339 M


Loop statement

The loop statements in awk are also used in C language and support while, do/while, for, break, and continue. These keywords have the same semantics as those in C language.



Because the subscript of an array in awk can be numbers and letters, the subscript of an array is usually called a key ). Both values and keywords are stored in an internal table that uses hash for key/value applications. Because hash is not stored in sequence, you will find that the array content is not displayed in the expected order. Arrays and variables are automatically created when they are used, and awk automatically determines whether they are stored as numbers or strings. In general, arrays in awk are used to collect information from records. They can be used to calculate the sum, count words, and track the number of times the template is matched.


Show/etc/passwd account


Awk-F': ''BEGIN {count = 0 ;}{ name [count] = $1; count ++ ;}; END {for (I = 0; I <NR; I ++) print I, name [I]} '/etc/passwd 0 root 1 daemon 2 bin 3 sys 4 sync 5 games


Java enterprise-level general permission security framework source code SpringMVC mybatis or hibernate + ehcache shiro druid bootstrap HTML5

[Download java framework source code]

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.