The SED of Shell scripting learning

In the process of writing shell scripts, we often need to process text files using the SED flow editor and awk. First, what is SED?

Sed is an online editor that processes a single line of content at a time. SED is a non-interactive editor. It does not modify the file unless you use Shell redirection to save the results. By default, all output lines are printed to the screen.

Ii. the process of SED

The SED editor processes the file (or input) row by line and sends the results to the screen. The process is as follows: first SED saves the row currently being processed in a temporary buffer (also known as pattern space), and then processes the rows in the temporary buffers and sends the line to the screen when it is complete. Sed deletes a row from the temporary buffer after each processing, and then reads the next line in, processing, and displaying it. After the last line of the input file has been processed, sed finishes running.

Before we say that SED does not modify the file, now we can know why? is because SED has a temporary buffer in each row and edits the copy, so the original file is not modified.

Additional knowledge:

In the process of using SED, we often hear "addressing", so what is "addressing"?

Addressing is used to decide which rows to edit. The address can be in the form of a number, a regular expression, or a combination of both. If no address is specified, SED processes all lines of the input file.

1. The address is a number that represents the line number; the "$" symbol indicates the last row.

For example:

[Plain]View Plaincopy

1. Sed-n ' 3p ' datafile #只打印第三行

2. Display only the file contents of the specified line range

For example:

[Plain]View Plaincopy

1. Sed-n ' 100,200p ' mysql_slow_query.log # View only lines 100th through No. 200 of the file

3, the address is separated by commas, then the address to be processed is the range between the two lines (including these two lines). Ranges can be represented by numbers, regular expressions, or a combination of both.

For example:

[Plain]View Plaincopy

1. Sed ' 2,5d ' datafile
2. #删除第二到第五行
4. Sed '/my/,/you/d ' datafile
5. Rows #删除包含 "My" to rows that contain "you"
7. Sed '/my/,10d ' datafile
8. #删除包含 the contents of line to line tenth of "My"

Iii. sed commands and options

The sed command tells SED how to handle each input line specified by the address, and all input lines are processed if no address is specified.

1. SED command

Command	Function
A\	Adds one or more rows after the current line. Multiple rows, except for the last row, need "\" to continue at the end of each line
C\	Replaces the text in the current line with the new text after this symbol. Multiple rows, except for the last row, need "\" to continue at the end of each line
I\	Inserts text before the current line. Multiple rows, except for the last row, need "\" to continue at the end of each line
D	Delete Row
H	Copy the contents of the schema space to the staging buffer
H	Append the contents of the schema space to the staging buffer
G	Copy the contents of the staging buffer to the schema space, overwriting the original content
G	Append the contents of the staging buffer to the schema space, appended to the original content
L	List non-printable characters
P	Print Line
N	Read into the next input line and start processing it from the next command instead of the first command
Q	End or exit sed
R	Reading input rows from a file
!	Apply commands to all rows except the selected row
S	Replace one string with another
G	Global substitution within a row
W	Writes the selected line to a file
X	Swap the contents of the staging buffer with the pattern space
Y	Replace a character with another character (you cannot use the Y command on a regular expression)

2. SED options

Options	Function
-E	Multiple edits, which are used when multiple SED commands are applied to an input line
-N	Cancel the default output
-F	Specify the file name of the SED script

Four, the regular expression meta-character

Like grep, SED also supports special meta-characters for pattern lookups and substitutions. The difference is that the regular expression used by SED is the pattern enclosed by the slash line "/".

If you want to change the regular expression delimiter "/" to another character, such as O, simply add a backslash before the character, followed by the regular expression after the character, followed by the character.

For example:

[Plain]View Plaincopy

1. Sed-n ' \o^myop ' datafile

The usual regular expressions are as follows:

Metacharacters	Function	Example
^	Beginning of Line Locator	/^my/matches all lines that begin with my
$	End-of-line locator	/my$/matches all lines ending with my
.	Match a single character other than the line break	/M.. y/matches the line that contains the letter m, followed by two arbitrary characters, and then the letter Y
*	Match 0 or more leading characters	/my*/matches a line containing the letter m followed by 0 or more Y-letters
[]	Matches any character within a specified character group	/[mm]y/matching rows that contain my or my
[^]	Matches any character that is not within the specified character group	/[^mm]y/matches the line that contains Y, but before y, the character is not m or m
..	To save a matched character	1,20s/youself/\1r/The pattern between the token metacharacters and saves it as a label 1, which can then be referenced by using \1. You can define up to 9 labels, numbering from the left, and the first one on the left. In this example, rows 1th through 20th are processed, you are saved as label 1, and if youself is found, it is replaced with your.
&	Save the lookup string to reference in the replacement string	s/my/**&**/Symbol & representative Find string. My will be replaced with **my**
\<	Word Head Locator	/\<my/matches lines that contain words that begin with my
\>	Suffix Locator	/my\>/matches a line that contains a word ending with my
X\{m\}	Continuous m x	/9\{5\}/matches a row containing 5 consecutive 9
X\{m,\}	At least a m x	/9\{5,\}/matches a row that contains at least 5 consecutive 9
X\{m,n\}	At least m, but not more than n x	/9\{5,7\}/matches rows that contain 5 to 7 consecutive 9

V. Exit status of SED

SED does not like grep, regardless of whether the specified pattern is found, its exit status is 0. The exit status of SED is not 0 only if there is a syntax error in the command.

Vi. examples of common use

1. P command

The command p is used to display the contents of the pattern space. By default, sed prints the input lines on the screen, and option-n is used to cancel the default print operation. When the Select-N and command p appear simultaneously, sed prints the selected content.

Example:

[Plain]View Plaincopy

1. (1) Sed '/my/p ' datafile
2. #默认情况下, sed prints all input lines on the standard output. If a row matches the pattern, the my,p command will print the line again.
4. (2) sed-n '/my/p ' datafile
5. #选项-N Cancels the default print for SED, and the p command prints the line of the matching mode my.

2, D command

Command d is used to delete input rows. Sed first copies the input line from the file into the pattern space, then executes the SED command on the line, and finally displays the contents of the pattern space on the screen. If command d is emitted, the input lines in the current mode space are deleted and not displayed.

Example:

[Plain]View Plaincopy

1. (1) Sed ' $d ' datafile
2. #删除最后一行, the rest is displayed.
4. (2) Sed '/my/d ' datafile
5. #删除包含my的行, the rest is displayed.

3, S command

Example:

[Plain]View Plaincopy

1. (1) Sed ' s/^my/you/g ' datafile
2. #命令末端的g表示在行内进行全局替换, that is, if a row appears with multiple my, all my my is replaced with you.
4. (2) sed-n ' 1,20S/MY$/YOU/GP ' datafile
5. #取消默认输出, Process 1 to 20 lines matching the line ending with my, replace all of my in-line I with you and print to the screen.
7. (3) Sed ' s#my#your#g ' datafile
8. #紧跟在s命令后的字符就是查找串和替换串之间的分隔符. Separating defaults is considered a forward slash, but can be changed. No matter what character (except newline, backslash), as soon as the S command is followed, a new string delimiter is formed.

4. E option

-E is an edit command that is used in cases where SED performs multiple editing tasks. All edit actions are applied to the line in the pattern buffer before the next line begins editing.

Example:

[Plain]View Plaincopy

1. Sed-e ' 1,10d '-e ' s/my/your/g ' datafile
2. #选项-E is used for multiple edits. The first edit deletes the 第1-3 line. The second editor replaces all occurrences of my with your. Because these two edits are performed line by row (that is, both commands are executed on the current line in the pattern space), the order in which the commands are edited affects the result.

5. R command

The r command is a read command. SED uses this command to add the contents of a text file to a specific location in the current file.

For example:

[Plain]View Plaincopy

1. Sed '/my/r introduce.txt ' datafile
2. #如果在文件datafile的某一行匹配到模式My, the contents of the file Introduce.txt are read after the line. If there is more than one row of my rows, the contents of the Introduce.txt file are read after the lines of my are present.

6, W command

Example:

[Plain]View Plaincopy

1. Sed-n '/hrwang/w me.txt ' datafile

7. A\ command

The A\ command is an append command, appended with the addition of new text to the current line in the file (that is, the line in the read-in mode buffer). The appended line of text is placed below the SED command. If more than one row is to be appended, each row must end with a backslash, except for the last line. The last line ends with a quotation mark and a file name.

Example:

[Plain]View Plaincopy

1. Sed '/^hrwang/a\
2. >hrwang and Mjfan are husband\
3. >and Wife ' DataFile
4. #如果在datafile文件中发现匹配以hrwang开头的行, append Hrwang and Mjfan are husband and wife below the line

8. I\ command

The I\ command inserts a new text in front of the current line.

9. c\ command

SED uses this command to modify an existing text to a new text.

10, N command

SED uses this command to get the next line of input files and read them into the pattern buffer, and any sed command will be applied to the next line immediately following the matching line.

For example:

[Plain]View Plaincopy

1. Sed '/hrwang/{n;s/my/your/;} ' datafile

Note: If you need to use multiple commands, or if you need to nest addresses in an address range, you must enclose the command in curly braces, write only one command per line, or use semicolons to split multiple commands in the same row.

11, y command

This command is similar to the TR command in Unix/linux, where characters are converted from left to right in a one-to-one fashion. For example, y/abc/abc/will convert all lowercase a to a, lowercase B to B, and lowercase C to c.

For example:

[Plain]View Plaincopy

1. Sed ' 1,20y/hrwang12/hrwang^$/' datafile
2. #将1到20行内, convert all lowercase hrwang to uppercase, convert 1 to ^, convert 2 to $.
3. #正则表达式元字符对y命令不起作用. As with the S command delimiter, the slash can be replaced with other characters.

12. Q Command

The Q command will cause the SED program to exit without further processing.

[Plain]View Plaincopy

1. Sed '/hrwang/{s/hrwang/hrwang/;q;} ' datafile

13, h command and G command

To better illustrate these two commands, we first create the following text file:

[Plain]View Plaincopy

1. #cat datafile
2. My name is Hrwang.
3. Your name is Mjfan.
4. Hrwang is Mjfan ' s husband.
5. Mjfan is Hrwang ' s wife.

[Plain]View Plaincopy

1. Sed-e '/hrwang/h '-e ' $G ' datafile
2. Sed-e '/hrwang/h '-e ' $G ' datafile
3. #通过上面两条命令, you will find that H clears the contents of the original staging buffer, saving only the schema space that was saved in the last time the H was executed. The h command appends the rows of each match Hrwnag to the staging buffer.
5. Sed-e '/hrwang/h '-e ' $g ' datafile
6. Sed-e '/hrwang/h '-e ' $G ' datafile
7. #通过上面两条命令, you'll find that G replaces the contents of the current row in the schema space with the content of the staging buffer, replacing the last row. The G command appends the contents of the staging buffer to the current line of the schema space. This is appended to the end.

Supplementary Knowledge Points: sed special usage

[Plain]View Plaincopy

1. Sed-n '/root/w a.txt ' #将匹配行输出到文件
3. Sed '/root/r abc.txt '/etc/passwd #把abc. txt file contents after reading into the root matching line
5. Sed-n '/root/w a.txt '
7. Sed-n '/root/{=;p} '/etc/passwd #打印行号和匹配root的行
9. Sed-n '/root/{n;d} '/etc/passwd #将匹配root行的下一行删除
11. Sed-n '/root/{n;d} '/etc/passwd #将匹配root行和下一行都删除
13. Sed ' 22{h;d};23,33{h;d};44g ' pass

Vii. sed Script Writing method

1. Read the command from the file

[Plain]View Plaincopy

1. Sed-f sed.sh

sed.sh File Contents:

[Plain]View Plaincopy

1. s/root/yerik/p
2. s/bash/csh/p

2. Run the script directly./sed.sh/etc/passwd

[Plain]View Plaincopy

1. #!/bib/sed-f
2. s/root/yerik/p
3. s/bash/csh/p

Eight, small tricks

1. Export your IP address with SED

[Plain]View Plaincopy

1. ifconfig eth0 |sed ' 2p ' |sed ' s/^.*addr://g ' |sed ' s/b.*$//g '

2. Use double quotes instead of the usual single quotation marks when referencing shell variables in the command line of SED. The following is a script that deletes the zone segment in the named.conf file according to the contents of the name variable:

Name= ' zone\ "localhost"

[Plain]View Plaincopy

1. Sed "/$name/,/};/d" named.conf

3. Hold and get: H command and G command

[Plain]View Plaincopy

1. $ Sed-e '/test/h '-e ' $G example

When the SED processes the file, each row is stored in a temporary buffer called the pattern space, unless the row is deleted or the output is canceled, otherwise all processed rows will be printed on the screen. The pattern space is then emptied and a new line is stored for processing. In this example, when a row matching test is found, it is stored in the pattern space, and the H command copies it into a special buffer called the hold cache. The second statement means that when the last line is reached, the G command takes out the row holding the buffer, then puts it back into the pattern space and appends it to the end of the line that already exists in the pattern space. In this example, it is appended to the last line. Simply put, any line containing test is copied and appended to the end of the file.

4. Hold and Swap: H command and x command

[Plain]View Plaincopy

1. $ Sed-e '/test/h '-e '/check/x ' example

Swaps the pattern space and preserves the contents of the buffer. That is, the line that contains test and check is swapped.

Nine, practice

1. Delete the first character of each line of the file.

1. Sed-n ' S/^.//GP '/etc/passwd
2. Sed-nr ' s/(.) (. *)/\2/p '/etc/passwd

2. Delete the second character of each line of the file.

1. Sed-nr ' s/(.) (.) (. *)/\1\3/p '/etc/passwd

3. Delete the last character of each line of the file.

1. Sed-nr ' s/.$//p '/etc/passwd
2. Sed-nr ' s/(. *) (.) /\1/p '/etc/passwd

4. Delete the second-to-last character of each line of the file.

1. Sed-nr ' s/(. *) (.) (.) /\1\3/p '/etc/passwd

5. Delete the second word in each line of the file.

1. Sed-nr ' s/([^a-z]*) ([a-z]+) ([^a-z]+] ([a-z]+) (. *)/\1\2\3\5/p '/etc/passwd

6. Delete the second-to-last word in each line of a file.

1. Sed-nr ' s/(. *) ([^a-z]+) ([a-z]+) ([^a-z]+] ([a-z]+) ([^a-z]*]/\1\2\4\5\6/p '/etc/samba/smb.conf

7. Delete the last word in each line of the file.

1. Sed-nr ' s/(. *) ([^a-z]+) ([a-z]+) ([^a-z]*]/\1\2\4/p '/etc/samba/smb.conf

8, swaps the first character and the second character of each line.

1. Sed-nr ' s/(.) (.) (. *)/\2\1\3/p '/etc/passwd

9, swap the first word and the second word for each line.

1. Sed-nr ' s/([^a-z]*) ([a-z]+) ([^a-z]+] ([a-z]+) (. *)/\1\4\3\2\5/p '/etc/samba/smb.conf

10, swap the first word and the last word of each line.

1. Sed-nr ' s/([^a-z]*) ([a-z]+) ([^a-z]+] ([a-z]+) (. *)/\1\4\3\2\5/p '/etc/passwd

11, delete all the numbers in a file.

1. Sed ' s/[0-9]*//g '/etc/passwd

12, remove all spaces at the beginning of each line.

1. Sed-n ' s/^\ *//p '/etc/samba/smb.conf
2. Sed-nr ' s/(*) (. *)/\2/p ' TESTP

13, replace any spaces that appear in the file with tabs.

1. Sed-n ' s/\/\t/gp ' pass

14, enclose all uppercase letters in parentheses ().

1. Sed-nr ' s/([A-z])/(&)/GP ' TESTP
2. Sed-n ' s/[a-z]/(&)/GP ' TESTP

15, print 3 times per line.

1. Sed ' p;p ' pass

16, interlace Delete.

1. Sed-n ' 1~2p ' Pass

17, copy the file from line 22nd to line 33rd to the end of line 44th.

1. Sed ' 1,21h;22h;23,33h;44g ' pass

18, move the file from line 22nd to line 33rd to the end of line 44th.

1. Sed ' 22{h;d};23,33{h;d};44g ' pass

19, only the first word of each line is displayed.

1. Sed-nr ' s/([^a-z]*) ([a-z]+) ([^a-z]+] (. *)/\2/p '/etc/passwd

20, print the first word and the third word of each line.

1. Sed-nr ' s/([^a-z]*) ([a-z]+] ([^a-z]+) ([a-z]+] ([^a-z]+) ([a-z]+] (. *)/\2--\4/p '/etc/passwd

21, change the format of the date format mm/yy/dd to MM;YY;DD

1. Date +%m/%y/%d |sed-n ' s#/#; #gp '

22, Reverse output

1. Cat A.txt
2. Abc
3. Def
4. Xyz
5. The output style becomes
6. Xyz
7. Def
8. Abc

The SED of Shell scripting learning