Linux. Shell programming notes-stream edit Sed

Chapter 7 stream editing what is sed selection Editor

The UNIX/Linux world has many text editors to choose from. For example, the most commonly used VI and emacs. With the most familiar editing tool, you can easily manage and edit tasks in UNIX.

Editors such as VI and emacs are called interactive editors. The interactive editor is great, but it does not help when we need to complete text processing in the program. In this case, you need some editing tools that can be completed in the command line.

We expect that all management processes will be automated, including editing files in batches. Many text editing needs to perform the same operation on each line of the text. This processing can be done with sed.

Sed is called a stream editor. What is a stream editor? The stream editor can edit the data received from standard input such as the MPs queue. Therefore, you do not need to store the data to be edited in files on the disk. Because data pipelines can be easily output to sed, it is easy to use sed as a long and complex pipeline in a powerful shell script.

Sed version

View version

[houchangren@ebsdi-23260-oozie ~]$ sed--versionGNU sed version 4.1.5Copyright (C) 2003 Free SoftwareFoundation, Inc.This is free software; see the source forcopying conditions.  There is NOwarranty; not even for MERCHANTABILITY orFITNESS FOR A PARTICULAR PURPOSE,to the extent permitted by law.

How Sed instances work

Sed executes any number of user-specified edit operations (commands) on the input data ). Sed is Row-based, so

Execute commands on each line in sequence. Sed then writes the result to the standard output (stdout) without modifying any input.

File.

For the parameter list, see:

Http://www.cnblogs.com/edwardlost/archive/2010/09/17/1829145.html

Example:

[houchangren@ebsdi-23260-oozie shell]$ head-n5 /etc/passwd > /tmp/passwd.bak[houchangren@ebsdi-23260-oozie shell]$cat  /tmp/passwd.bakroot:x:0:0:root:/root:/bin/bashbin:x:1:1:bin:/bin:/sbin/nologindaemon:x:2:2:daemon:/sbin:/sbin/nologinadm:x:3:4:adm:/var/adm:/sbin/nologinlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin[houchangren@ebsdi-23260-oozie shell]$ sed-e 'd'  /tmp/passwd.bak[houchangren@ebsdi-23260-oozie shell]$ sed-e '1d' /tmp/passwd.bakbin:x:1:1:bin:/bin:/sbin/nologindaemon:x:2:2:daemon:/sbin:/sbin/nologinadm:x:3:4:adm:/var/adm:/sbin/nologinlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin[houchangren@ebsdi-23260-oozie shell]$ sed-e '3d' /tmp/passwd.bakroot:x:0:0:root:/root:/bin/bashbin:x:1:1:bin:/bin:/sbin/nologinadm:x:3:4:adm:/var/adm:/sbin/nologinlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

The 'd' above is the parameter application. By default, it is used to delete each row. When a number is specified, the records in the first row are deleted. Delete 3rd items for the second time. The remaining 4 items are displayed.

NOTE

In this example, note the following:

1) no modification was made to/tmp/passwd. bak. This is because sed only reads the file specified in the command line and uses it as the input-it does not try to modify the file.

2) It should be noted that sed is Row-oriented. The 'd' command does not simply tell sed to delete all input data at once. Conversely, sed lines by line/etc/passwd. each row of a bak is read into the internal buffer called the mode buffer. Once a row is read into the mode buffer, it runs the 'D' command, then print the content of the mode buffer (NO content in this example ). if the address is not used, the command applies to all rows.

3) enclose the usage of the 'd' command in single quotes. It is a good idea to develop the habit of using single quotes to enclose the scd command, so that shell extension can be disabled.

Sed address range

Specify the operation line area of the command, such as the following 1-2 lines and 3-6 lines. Of course, if there are no 6th lines, only how much is deleted.

[Houchangren @ ebsdi-23260-oozie shell] $ sed-e '1, 2d '/tmp/passwd. bak
Daemon: x: 2: 2: daemon:/sbin/nologin
Adm: x: 3: 4: adm:/var/adm:/sbin/nologin
Lp: x: 4: 7: lp:/var/spool/lpd:/sbin/nologin
[Houchangren @ ebsdi-23260-oozie shell] $ sed-e '3, 6d '/tmp/passwd. bak
Root: x: 0: 0: root:/bin/bash
Bin: x: 1: 1: bin:/sbin/nologin

Sed ignore comments

[houchangren@ebsdi-23260-oozie shell]$ cat/etc/rc.local#!/bin/sh## This script will be executed *after* allthe other init scripts.# You can put your own initialization stuffin here if you don't# want to do the full Sys V style initstuff. touch /var/lock/subsys/localhive --service hiveserver &[houchangren@ebsdi-23260-oozie shell]$ sed-e '/^#/d' /etc/rc.local | more touch /var/lock/subsys/localhive --service hiveserver & [houchangren@ebsdi-23260-oozie shell]$ sed-e '/^[^#]/d' /etc/rc.local | more#!/bin/sh## This script will be executed *after* allthe other init scripts.# You can put your own initialization stuffin here if you don't# want to do the full Sys V style initstuff.

Rule expression

Rule expression characters used in Sed

Character	Description
^	Match with the beginning of the line
$	Match with the end of a row
.	Match any character
*	Matches zero or multiple of the previous characters.
[]	Matches all characters in [].

Sed rule expression instance

Rule expression	Description
/./	Match any row containing at least one character
/../	Match any line containing at least two characters
/^ #/	It will match any row starting with '#'. This is usually a comment.
/} $/	Match any row ending '}'
/} * $/	Note that there is a space behind}, which will match with any row that ends with zero or multiple spaces after '}'
/[Abc]/	Match any row containing lowercase letters 'a, B, and c
/^ [Abc]/	Will match any row starting with a, B, c

The-n parameter indicates that sed does not do this unless the print mode space is explicitly required.

[houchangren@ebsdi-23260-oozie shell]$ sed-n -e '/^[echo]/p' user_login.shecho " user $1 is on"elseecho " user $1 is off" [houchangren@ebsdi-23260-oozie shell]$ sed-n -e '/[abc]/p' user_login.sh#!/bin/bashfunction user_login(){echo " user $1 is on"echo " user $1 is off" [houchangren@ebsdi-23260-oozie shell]$ cata.c#include <stdio.h>#include <math.H> int main (){// int base,n;// scanf("%d,%d\n",&b,&n) }[houchangren@ebsdi-23260-oozie shell]$ sed-n -e '/main[[:space:]]*(/,/^ }/p' a.c | moreint main (){// int base,n;// scanf("%d,%d\n",&b,&n) }[houchangren@ebsdi-23260-oozie shell]$ sed-n -e '/^\//p' a.c | more// int base,n;// scanf("%d,%d\n",&b,&n) 

Powerful Sed function replacement

Replacement Formula: sed-e s [Symbol] [character to be replaced] [Symbol] [character after replacement] [Symbol] [g]

Instance:

[Houchangren @ ebsdi-23260-oozie data] $ cattwister.txt // view a message I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. [houchangren @ ebsdi-23260-oozie data] $ sed-e's/wish/want/'twister.txt // Replace the first matching I want to wish the wish you wish to wish, but if you wish the witch wantes, I won't wish the wish you wish towish. I want to wish the wish you wish to wish, but if you wish the witch wantes, I won't wish the wish you wish towish. I want to wish the wish you wish to wish, but if you wish the witch wantes, I won't wish the wish you wish towish. I want to wish the wish you wish to wish, but if you wish the witch wantes, I won't wish the wish you wish towish. [houchangren @ ebsdi-23260-oozie data] $ sed-e's/wish/want/G' twister.txt // replace all I want to want the want you want to want, but if you want the witch wantes, I won't want the want you want towant. I want to want the want you want to want, but if you want the witch wantes, I won't want the want you want towant. I want to want the want you want to want, but if you want the witch wantes, I won't want the want you want towant. I want to want the want you want to want, but if you want the witch wantes, I won't want the want you want towant. [houchangren @ ebsdi-23260-oozie data] $ sed-e '1, 2 s/wish/want/'twister.txt // Replace the first matching I want to wish the wish you wish to wish, but if you wish the witch wantes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. [houchangren @ ebsdi-23260-oozie data] $ sed-e '1, 2 s/wish/want/G' twister.txt // replace all matching I want to want the want you want to want, but if you want the witch wantes, I won't want the want you want towant. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. I wish to wish the wish you wish to wish, but if you wish the witch wishes, I won't wish the wish you wish towish. [houchangren @ ebsdi-23260-oozie data] $

Because sometimes you need to operate the '/' symbol, use the '/' line again.

It is inconvenient to use the symbol. You can change it to another one, such as the colon ':'

[houchangren@ebsdi-23260-oozie data]$ cathivepath.txt/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr/local/hive-0.7.1-cdh3u6/bin/hive[houchangren@ebsdi-23260-oozie data]$ sed-e s:/usr/local:/usr/lib:g hivepath.txt/usr/lib/hive-0.7.1-cdh3u6/bin/hive/usr/lib/hive-0.7.1-cdh3u6/bin/hive/usr/lib/hive-0.7.1-cdh3u6/bin/hive/usr/lib/hive-0.7.1-cdh3u6/bin/hive

Filter all html Tag instances

[houchangren@ebsdi-23260-oozie data]$ cattest.html<!DOCTYPE HTML PUBLIC "-//W3C//DTDHTML 4.0 Transitional//EN"><HTML> <HEAD> <TITLE> New Document </TITLE> <META NAME="Generator" CONTENT="EditPlus"> <META NAME="Author" CONTENT=""> <META NAME="Keywords" CONTENT=""> <META NAME="Description" CONTENT=""> </HEAD>  <BODY>  I'mtest html </BODY></HTML> [houchangren@ebsdi-23260-oozie data]$ sed-e 's/<[^>]*>//g' test.html New DocumentI'mtest html

Combine multiple commands

You can use the Semicolon ";" when using multiple commands at the same time.

[Houchangren @ ebsdi-23260-oozie data] $ sed-e '= 'fruit.txt // equal sign = is the display row number 1% % banae 2 banana 3apple4Presimmon5% banae 6 apple 7 Banana 8 orange 9 presimmon [houchangren @ ebsdi-23260-oozie data] $ sed-e' =; p'fruit.txt // display the row number and print it. The default value is 1% % banae 2 banana 3appleapple4PresimmonPresimmon5% banae 6 apple 7 Banana banana 8 orange 9 presimmonpresimmon [houchangren @ ebsdi-23260-oozie data] $ sed-n-e '=; p'fruit.txt // print the p Parameter command after specifying-n 1% % banae 2 banana 3apple4Presimmon5% banae 6 apple 7 Banana 8 orange 9 Persimmon [houchangren @ ebsdi-23260-oozie data] $ sed- n-e 'P'-e = fruit.txt // you can specify multiple commands. % banae 1 banana 2apple3Presimmon4% banae 5 apple 6 Banana 7 orange 8presimmon9

Sometimes, when too many commands need to be connected and executed together,-e may not be enough. You can use-f to specify the text and then write the command in the text.

[Houchangren @ ebsdi-23260-oozie data] $ cathivepath.txt/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr/local/hive-0.7.1-cdh3u6/bin/hive/usr /local/hive-0.7.1-cdh3u6/bin/hive // 1d is to delete a row, then replace/usr/local with/usr/bin, print, and display the row number [houchangren @ ebsdi-23260-oozie data] $ cat .. /sed/test. sed 1ds:/usr/local/:/usr/lib/: gp = [houchangren @ ebsdi-23260-oozie data] $ sed-n-f .. /sed/test. sed hivepath.txt/usr/lib/hive-0.7.1-cdh3u6/bin/hive2/usr/lib/hive-0.7.1-cdh3u6/bin/hive3/usr/lib/hive-0.7.1-cdh3u6/bin/hive4

Apply multiple commands to one address range

Specify an address range, such as 1-5 rows, and then perform multiple operations.

[houchangren@ebsdi-23260-oozie data]$ head-n10 /etc/passwd > pwd.piece[houchangren@ebsdi-23260-oozie data]$ catpwd.pieceroot:x:0:0:root:/root:/bin/bashbin:x:1:1:bin:/bin:/sbin/nologindaemon:x:2:2:daemon:/sbin:/sbin/nologinadm:x:3:4:adm:/var/adm:/sbin/nologinlp:x:4:7:lp:/var/spool/lpd:/sbin/nologinsync:x:5:0:sync:/sbin:/bin/syncshutdown:x:6:0:shutdown:/sbin:/sbin/shutdownhalt:x:7:0:halt:/sbin:/sbin/haltmail:x:8:12:mail:/var/spool/mail:/sbin/nologinnews:x:9:13:news:/etc/news:[houchangren@ebsdi-23260-oozie data]$ sed -n -e'1,5{s:/bin/bash:/bin/sh:g;s/:/|/g;p}' pwd.piece[houchangren@ebsdi-23260-oozie data]$ sed-n -e '1,5{s:/bin/bash:/bin/sh:g;s/:/|/g;p}' pwd.pieceroot|x|0|0|root|/root|/bin/shbin|x|1|1|bin|/bin|/sbin/nologindaemon|x|2|2|daemon|/sbin|/sbin/nologinadm|x|3|4|adm|/var/adm|/sbin/nologinlp|x|4|7|lp|/var/spool/lpd|/sbin/nologin[houchangren@ebsdi-23260-oozie data]$ [houchangren@ebsdi-23260-oozie data]$ cat../sed/pwd.sed1,5{s:/bin/bash:/bin/sh:gs/:/|/gp}[houchangren@ebsdi-23260-oozie data]$ sed-n -f ../sed/pwd.sed pwd.pieceroot|x|0|0|root|/root|/bin/shbin|x|1|1|bin|/bin|/sbin/nologindaemon|x|2|2|daemon|/sbin|/sbin/nologinadm|x|3|4|adm|/var/adm|/sbin/nologinlp|x|4|7|lp|/var/spool/lpd|/sbin/nologin

Instance example

LINUX and DOS/Windows systems use different line breaks in plain text format. This script converts UNIX-style text to DOS/windows format. As you may know, DOS and windows-based text files have a CR (Press ENTER) and LF (line feed) at the end of each line, while UNIX text has only one line break. Sometimes you may need to move a unix text

In Windows, the script will perform the necessary format conversion for you.

>>> sed  -e  ‘s/$/\r/’myunix.txt > mydos.txt

In this script, the '$' rule expression matches the end of the row, and '\ R' tells sed to insert a carriage return before it.

Insert a carriage return before line feed, and each line ends with CR/LF immediately.

On the contrary, many times the downloaded network files are dos/windows files, which are indeed problematic in unix operations, such as bash. Use sed to convert dos/windows text to a trusted unix format

>>> sed –e ‘s/.$//’ mydos.txt >myunix.txt

The script works very easily: the replacement rule expression matches the last character of a row, and the character is exactly the carriage return. We can replace it with an empty character to completely delete it from the output. If you use the script change and notice that the last character of each line in the output has been deleted, you specify a UNIX text file.

Sed practice

Requirements:

1. Change "James" to "Li Xiaoming ";

2. Delete the first three rows

3. Display 5-10 rows

4. Delete the row containing "exclude"

5. display all rows with birthdays between Nov and Dec

6. All rows starting with a surname Zhang, marked with the front side ***

7. Use "trial staff" to replace the line containing "trial"

8. The birthday invented by Liu was 1986/11/11

9. Delete all blank rows

10. write a script to insert the first line into the Personnel File, delete all salaries ending with 500, display the File content, turn the phone number and birthday upside down, and add the end at the end of the File.

11. Extract the name of the row containing "trial" and replace it with "trial User:" + name

Data content:

[Houchangren @ ebsdi-23260-oozie data] $ cat persons.txt James: 010-68239343: I am a manager: 1988/01/10: 5000 Dawn: 010-68239343: I am a manager: 1988/08/10: 5000 Zhang Xueyou: 010-68239343: I am a manager: 1988/04/10: 5000 Zhou Yunfei: 010-68239343: My business has been ruled out.: 1988/10/10: 500 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 great wilderness West: 010-68239343: no description: 1988/10/10: 1000 STEPHEN: 010-68239343: no description: 1988/10/10: 1000 large: 010-68239343: trial.: 1988/10/10: 200 task operation: 1. [houchangren @ ebsdi-23260-oozie data] $ sed-n-e's: James: Li Xiaoming: gp 'persons.txt Li Xiaoming: 010-68239343: I am a manager: 1988/01/10: 50002. [houchangren @ ebsdi-23260-oozie data] $ sed-e '1, 3d 'persons.txt Zhou Yunfei: 010-68239343: The service is excluded.: 1988/10/10: 500 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 great wilderness West: 010-68239343: no description: 1988/10/10: 1000 STEPHEN: 010-68239343: no description: 1988/10/10: 1000 large: 010-68239343: trial.: 1988/10/10: 2003. zhou Yunfei: 010-68239343: the business has been ruled out.: 1988/10/10: 500 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 4. [houchangren @ ebsdi-23260-oozie data] $ sed-e '/exclude/d' persons.txt James: 010-68239343: I am a manager: 1988/01/10: 5000 Dawn: 010-68239343: I am a manager: 1988/08/10: 5000 Zhang Xueyou: 010-68239343: I am a manager: 1988/04/10: 5000 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 great wilderness West: 010-68239343: no description: 1988/10/10: 1000 STEPHEN: 010-68239343: no description: 1 988/10/10: 68239343 large: 010-: the trial is in progress.: 1988/10/10: 200 5. [houchangren @ ebsdi-23260-oozie data] $ sed-n'/[:] [0-9] * [:/:] 1 [1-2]/P' persons.txt lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 53406. [houchangren @ ebsdi-23260-oozie data] $ sed-n's/^ Zhang/*** Zhang/p 'persons.txt *** Zhang Xueyou: 010-68239343: I am a manager: 1988/04/10: 5000 7. [houchangren @ ebsdi-23260-oozie data] $ sed-e's/^. * trial. * $/trial staff/G' persons. Txt James: 010-68239343: I am a manager: 1988/01/10: 5000 Dawn: 010-68239343: I am a manager: 1988/08/10: 5000 Zhang Xueyou: 010-68239343: I am a manager: 1988/04/10: 5000 Zhou Yunfei: 010-68239343: the business was ruled out.: 1988/10/10: 500 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 great wilderness West: 010-68239343: no description: 1988/10/10: 1000 STEPHEN: 010-68239343: no description: 1988/10/10: 1000 trial staff 8. [houchangren @ ebsdi-23260-oozie data] $ sed-n-e '/Liu invention/s/: [0-9] * \/. *\/. *:/: 1986 \/11 \/11:/gp 'persons.txt Liu invention: 010-68239343: no description: 1986/11/11: 53409. [houchangren @ ebsdi-23260-oozie data] $ sed-e '/^ $/d 'persons.txt James: 010-68239343: I am MANAGER: 19 88/01/AM: 010-68239343: I am a manager: 1988/08/10: 5000 Zhang Xueyou: 010-68239343: I am a manager: 1988/04/10: 5000 Zhou Yunfei: 010-68239343: The business is ruled out.: 1988/10/10: 500 lize card: 010-68239343: no description: 1988/12/10: 2000 Liu invention: 010-68239343: no description: 1988/11/10: 5340 great wilderness West: 010-68239343: no description: 1988/10/10: 1000 STEPHEN: 010-68239343: no description: 1988/10/10: 1000 large: 010-68239343: trial.: 1988/10/10: 200 10. [houchangren @ ebsdi-23260-oozie data] $ cat .. /sed/person. sed/500 $/ds /\(. *\)\(:. *:\)\(. *\)\(:. *:\)\(. * \)/\ 1 \ 4 \ 3 \ 2 \ 5/g1i personnel file $ a end [houchangren @ ebsdi-23260-oozie data] $ sed-f .. /sed/person. sed persons.txt personnel file James: 1988/01/10: I am a manager: 010-68239343: 5000 Dawn: 1988/08/10: I am a manager: 010-68239343: 5000 Zhang Xueyou: 1988/04/10: I am a manager: 010-68239343: 5000 lize card: 1988/12/10: no description: 010-68239343: 2000 Liu invention: 1988/11/10: no description: 010-68239343: 5340 great wilderness West: 1988/10/10: no description: 010-68239343: 1000 STEPHEN: 1988/10/10: no description: 010-68239343: 1000 large: 1988/10/10: the trial is in progress.: 010-68239343: 200 the end 11. [houchangren @ ebsdi-23260-oozie data] $ sed-n-e '/trial/s /\(. *\):. *:. *:. *:. */trial staff: \ 1/gp 'persons.txt | more trial staff: large quota