Sed & awk basic usage

Sed tool Introduction

After learning about some basic applications of formal notation, what about it? Haha ~ You can play with two things, that is, sed and awk! These two guys are quite useful! For example, the logfile. Sh written by laruence analyzes the small programs in the login file, and uses and counts most of the analysis keywords. These two baby eggs are used to help me! So you said, do you want to have fun ?! Pai_^

Let's talk about sed first. Basically, sed can analyze the data of standard input (stdin), and then output the data to standrad out (stdout) after processing). What about processing? You can replace, delete, add, and retrieve specific rows! Pretty good ~ Let's first understand the usage of SED, and then talk about its usage! [Root @ Linux ~] # Sed [-nefri] [action]

Parameters:

-N: silent mode. In general sed usage, all data from stdin is usually listed on the screen. However, if the-n parameter is added, only the row (or action) that has been specially processed by SED will be listed.

-E: directly edit the SED action in the Command column mode;

-F: Write the SED action directly in a file.-f filename can execute the SED action in filename;

-R: sed supports the syntax of extended regular notation. (The default is the basic regular expression syntax)

-I: directly modify the read file content, rather than the screen output.

Action Description: [N1 [, N2] Function

N1, N2: It may not exist. It generally indicates "number of rows selected for the action". For example, if my action needs to be performed between 10 and 20 rows, then "[Action Behavior]"

Function has the following functions:

A: new. A can be followed by strings. These strings will appear in the new row (the next row currently )~

C: replace. C can be followed by strings. These strings can replace rows between N1 and N2!

D: delete. Because it is deleted, D is usually followed by no comment;

I: insert, I can be followed by strings, and these strings will appear in the new line (the previous line currently );

P: print the selected data. Usually P will work with the sed-N parameter ~

S: replace, you can directly replace the work! Generally, this s action can be used together.

Regular representation! For example, 1, 20 s/old/new/g!

Example:

Example 1: List the content of/etc/passwd and print the row number ~ Delete 5 rows!

[Root @ Linux ~] # NL/etc/passwd | sed '2, 5d'

1 root: X: 0: 0: Root:/root:/bin/bash

6 Sync: X: 5: 0: Sync:/sbin:/bin/Sync

7 shutdown: X: 6: 0: shutdown:/sbin/Shutdown

... (Omitted later ).....

# Have you seen it? Because 2-5 rows are deleted from the table, NO 2-5 rows are displayed ~

# In addition, note that sed-E should have been issued, and no-E is required!

# At the same time, note that the action followed by sed must be enclosed in two single quotes!

# If you only need to delete 2nd rows, you can use NL/etc/passwd | sed '2d,

# For the last line from 3rd to, NL/etc/passwd | sed '3, $ d!

Example 2: Add "drink tea?" To the question behind the second line (that is, to the third line ?』 Words!

[Root @ Linux ~] # NL/etc/passwd | sed '2a drink tea'

1 root: X: 0: 0: Root:/root:/bin/bash

2 bin: X: 1: 1: Bin:/bin:/sbin/nologin

Drink tea

3 daemon: X: 2: 2: daemon:/sbin/nologin

# Hey hey! The string added after line A will already appear after line 2! What if it is before the second line?

# NL/etc/passwd | sed '2i drink tea 'is right!

Example 3: Add two lines after the second line, for example, "Drink tea or..." Drink beer ?』

[Root @ Linux ~] # NL/etc/passwd | sed '2a drink tea or ....../

> Drink beer? '

1 root: X: 0: 0: Root:/root:/bin/bash

2 bin: X: 1: 1: Bin:/bin:/sbin/nologin

Drink tea or ......

Drink beer?

3 daemon: X: 2: 2: daemon:/sbin/nologin

# The focus of this example is that we can add more than one line! Several lines can be added ~

# However, each row must be added with a backslash! Therefore, in the above example,

# We can find that there is/at the end of the first line! That must be done!

Example 4: I want to replace the content in line 2-5 with "NO 2-5 number?

[Root @ Linux ~] # NL/etc/passwd | sed '2, 5C NO 2-5 number'

1 root: X: 0: 0: Root:/root:/bin/bash

No 2-5 Number

6 Sync: X: 5: 0: Sync:/sbin:/bin/Sync

# No rows 2-5, hey! The data we need will show up!

Example 5: only 5th-7 rows are listed.

[Root @ Linux ~] # NL/etc/passwd | sed-n'5, 7p'

5 LP: X: 4: 7: LP:/var/spool/lpd:/sbin/nologin

6 Sync: X: 5: 0: Sync:/sbin:/bin/Sync

7 shutdown: X: 6: 0: shutdown:/sbin/Shutdown

# Why do we need to add the-n parameter? You can issue sed '5, 7p' on your own! (Repeated output in rows 5-7)

# When the-n parameter is added, the output data is much worse!

Example 6: We can use ifconfig to list IP addresses?

[Root @ Linux ~] # Ifconfig eth0

Eth0 link encap: Ethernet hwaddr 00: 51: FD: 52: 9A: CA

Inet ADDR: 192.168.1.12 bcast: 192.168.1.255 mask: 255.255.255.0

Inet6 ADDR: fe80: 250: fcff: fe22: 9acb/64 scope: Link

Up broadcast running Multicast MTU: 1500 Metric: 1

... (Omitted below ).....

# Actually, all we need is the inet ADDR:... line. So, use grep and sed to catch it.

[Root @ Linux ~] # Ifconfig eth0 | grep 'inet '| SED's/^. * ADDR: // G' |/

> SED's/bcast. * $ // G'

# You can run the process of each pipeline (|) separately to find out the cause!

# After we start and end, we will get the IP address we need, that is, 192.168.1.12 ~

Example 7: extract the man settings in the/etc/man. config file, but do not describe the content.

[Root @ Linux ~] # Cat/etc/man. config | grep 'man '| SED's/#. * $/G' |/

> SED '/^ $/d'

# Sed '/^ $/d' is used to delete empty rows,

# If there is # In each row, it indicates the behavior annotation, but note that sometimes,

# Annotation is not written in the first character, that is, after a command, as shown below:

# "Shutdown-H now # This is the command to shut down", and the comment # is behind the command.

# Therefore, we will use the regular notation #. * $!

Example 8: Use sed to directly ~ /Add "# This is a test" to the last line of bashrc 』

[Root @ Linux ~] # Sed-I '$ A # This is a test '~ /. Bashrc

# The-I parameter in the header allows your sed to directly modify the file content next to it! Instead of screen output.

# The $ A indicates that the last row is added.

In short, this SED is a good use! In addition, many shell scripts use the function of this command ~ Sed can help the system administrator manage daily work! You need to study it carefully!

Introduction to awk tools

Compared with SED, awk tends to divide a row into several "fields" for processing. Therefore, awk is suitable for processing small data processing! Awk usually operates in the following mode: [root @ Linux ~] # Awk 'condition type 1 {Action 1} Condition Type 2 {Action 2}... 'filename

Awk can process subsequent files or read standard output from the previous command. As mentioned above, awk mainly processes "data in fields in each row", and the preset "field separator is" Space key "or" [Tab] Key "』! For example, we can use last to retrieve the data of the login user. The result is as follows: [root @ Linux ~] # Last

Dmtsai pts/0 192.168.1.12 Mon Aug 22 still logged in

Root tty1 Mon Aug 15)

Reboot system boot 2.6.11 sun Aug 14 (7 + 15: 41)

Dmtsai pts/0 192.168.1.12 Fri Aug 12)

If I want to retrieve the IP address of the account and the login user, and the account and IP address are separated by [Tab], it will become like this: [root @ Linux ~] # Last | awk '{print $1 usd/t "$3 }'

Dmtsai 192.168.1.12

Root mon

Reboot boot

Dmtsai 192.168.1.12

No matter which row I want to process, there is no need to limit the "Condition Type! What I want is the first and third columns, but the content in the second and third rows is strange ~ This is because of the data format problem! So ~ When using awk, please first confirm that your data contains continuity data without spaces or [tabs]. Otherwise, it will be like this example, misjudgment will occur!

In addition, you will also know from the above example that every field in each row has a variable name, that is, $1, $2... for example, dmtsai is $1 because it is the first column! 192.168.1.12 is the third column, so it is $3! And so on ~ Haha! There is another variable! That is, $0 and $0 represent the meaning of "a full column of data ~ In the preceding example, $0 in the first line represents the line "dmtsai pts/0! We can see that the entire awk processing process in the above four rows is:

Read the first line, and fill in the data in the first line with variables such as $0, $1, $2;

Determine whether to perform the subsequent "action" based on the "Condition Type" restriction ";

Complete all the actions and condition types;

If there is any subsequent "row" data, repeat the above 1 ~ Step 3 until all data is read.

After this step, you will know that awk is a "unit for processing a behavior", and "unit for processing the smallest field 』. Well, how does awk know how many rows of data I have? How many columns are there? This requires the help of the built-in variables of awk ~

Variable name meaning

Total number of fields owned by each line of NF ($0)

NR currently awk processes the "row number" Data

The delimiter currently used by fs. The default Delimiter is the Space key.

Let's continue with the above example. If I want to list the accounts for each row and list the number of currently processed rows, and specify the number of fields in the row, then (note, all subsequent actions of the awk are enclosed by '. Therefore, if you want to print the content in print, remember that the text section of the non-variable contains the format mentioned in printf in the previous section, double quotation marks are required for definition !) [Root @ Linux ~] # Last | awk '{print $1 usd/t lines: "Nr"/T columes: "NF }'

Dmtsai lines: 1 columes: 10

Root lines: 2 columes: 9

Reboot lines: 3 columes: 9

Dmtsai lines: 4 columes: 10

So we can understand the difference between NR and NF? Now let's talk about the so-called "condition type!

Logical operation character of awk

Now that you need to use the "condition" type, you need some logical operations ~ For example:

Computing Unit meaning

> Greater

<Less

> = Greater than or equal

<= Less than or equal

= Equal

! = Not equal

It is worth noting that the = symbol, because in the "logical operation", it is the so-called attention formula that is greater than, less than, equal to, etc., we are used to represent =, if a value is directly given, for example, when a variable is set, the value = is used directly. Well, let's use the logic to judge it! For example, in/etc/passwd, the fields are separated by colons ":". For example, if I want to check the data below 10 in the third column, and only list accounts and the third column. You can do this: [root @ Linux ~] # Cat/etc/passwd |/

> Awk '{FS = ":" }3 3 <10 {print $1 usd/t "$3 }'

Root: X: 0: 0: Root:/root:/bin/bash

Bin 1

Daemon 2

...... (Omitted below )......

Fun! However, why is the first line not displayed correctly? This is because when we read the first line, the variables $1, $2... the default value is still separated by the Space key, so although we have defined FS = ":", it takes effect only after the second line. So what should we do? We can set awk variables in advance! Use the begin keyword! In this way: [root @ Linux ~] # Cat/etc/passwd |/

> Awk 'in in {FS = ":"} $3 <10 {print $1 usd/t "$3 }'

Root 0

Bin 1

Daemon 2

...... (Omitted below )......

Interesting! Besides begin, we also have end! What if we want to use awk for the "computing function? In the following example, assume that I have a salary data table with the following content: Name 1st 2nd 3th

Vbird 23000 24000 25000

Dmtsai 21000 20000 23000

Bird2 43000 42000 41000

How can I calculate the total amount of each person? I also want to format the output! You can store the above data into a file named pay.txt, then: [root @ Linux ~] # Cat pay.txt |/

> Awk 'nr = 1 {printf "% 10 S % 10 S % 10 S % 10 S % 10 s/n", $1, $2, $3, $4, "Total "}

NR> = 2 {Total = $2 + $3 + $4

Printf "% 10 S % 10d % 10d % 10d % 10.2f/N", $1, $2, $3, $4, total }'

Name 1st 2nd 3th total

Vbird 23000 24000 25000 72000.00 dmtsai 21000 20000 23000 64000.00 bird2 43000 42000 41000 126000.00

The above example has several important things that should be explained first:

All actions, that is, actions within {}, can be separated by semicolons (;) if multiple instructions are needed, or directly use the [enter] key to separate each command. For example, the above Nr> = 2 followed by the action, use Total =... that command is used to specify the sum, while printf will be used to format the output later!

In the logic operation, if it is "equal to", you must use two equal signs "= 』!

When formatting the output, you must add/N in the format setting of printf to split it!

Unlike the bash shell variable, the variable can be used directly in the awk without adding the $ symbol.

Awk can help us deal with a lot of daily work! It's really easy to use ~ In addition, the output format of awk is often supported by printf. Therefore, it is better to familiarize yourself with printf! In addition, {} in the awk action also supports if (condition! For example, the preceding command can be changed to [root @ Linux ~]. # Cat pay.txt |/

> Awk '{If (Nr = 1) printf "% 10 S % 10 S % 10 S % 10 S % 10 s/n", $1, $2, $3, $4, "Total "}

NR> = 2 {Total = $2 + $3 + $4

Printf "% 10 S % 10d % 10d % 10d % 10.2f/N", $1, $2, $3, $4, total }'

You can carefully compare the two inputs above ~ Learn about the two syntaxes! I personally prefer to use the first syntax, because it will be more unified! Pai_^

Appendix: differences between print and printf in bash

Print outputs the specified content and then wrap it. printf only outputs the specified content and does not wrap it.

Awk '{for (I = 1; I <5; I ++) {printf ($ I "/t")}' Test

Awk '{for (I = 1; I <5; I ++) {print ($ I "/t")}' Test

What are the outputs of these two commands?

Test content:

M. tansley 05/99 48311 green 8 40 44

J. Lulu 06/99 48317 green 9 24 26

P. Bunny 02/99 4 8 yellow 12 35 28

J. Troll 07/99 4842 brown-3 12 26 26

L. tansley 05/99 4712 brown-2 12 30 28

Awk '{for (I = 1; I <5; I ++) {printf ($ I "/t")}' Test

M. tansley 05/99 48311 green J. Lulu 06/99 48317 green P. Bunny 02/99 48 yellow J. Troll 07/99 4842 brown-3 L. tansley 05/99 4712 brown-2

Awk '{for (I = 1; I <5; I ++) {print ($ I "/t")}' Test

M. tansley

05/99

48311

Green

J. Lulu

06/99

48317

Green

P. Bunny

02/99

48

Yellow

J. troll

07/99

4842

Brown-3

L. tansley

05/99

4712

Brown-2