Linux Shell's ten sed and awk

When you design script, you sometimes need to modify your scripts, such as deleting or replacing certain keywords. Like this practice of dynamically modifying files during script execution, called stream editing. A tool with flow editing capabilities, called the Flow editor. SED is the strong in this respect. In addition, script execution may have to make a report, rendering each field information, awk perfect solution.

First, regular expression

Regular expressions are the basic syntax for composing "styles," and "styles" are the necessary capabilities to use SED and awk. Sed and awk run the same way as long as the "style" data row is met, and the specified "action" is performed on it.

What is a regular expression?

A regular expression is a descriptive method, a small language that represents a combination of a style or a number of styles, the power of which is that it takes just a few simple characters to represent many strings in common.

1,. Represents any character

. A. Represents the middle of a, 3 characters of random characters on both sides. (To compare. You need to use \ Escape)

2. ^ represents at the beginning of the line

^abc Abd should appear at the top of the row. "Abc,hello" and "Hello ABC" the former conform to the latter does not conform

3, $ stands at the tail

$abc Abd should appear at the end of the line. "Abc,hello" and "Hello abc" the latter conform to the former does not conform

4, [...] character set

[ ...] represents a character in a string line [ABc] stands for a or B or C. [A-z] a uppercase [^a-z] character other than uppercase.

5, * more than 0

A*c can be ABC, ABBC, ABBBC, ASJKSKBKC .....

6, \{...\} Specify the number of matches

\{3,5\} There are three characters in front of it. [A-z]\{3,5\} represents a string of lowercase letters with a length of

7, \ (.. \) Save the matching strings for the time being.

a\ (.. \) b to save 2 characters between A and B, to extract the saved string, the available positional arguments, \1 represents the first saved string, and \2 represents the second persisted string.

Second, extended regular expression

Re character meanings and examples

+ Repeat 1 or more than 1 first re characters

Egrep ' go+d ' File search range is god good goood gooood ... such as

? 0 or 1 first re characters

Egrep ' go?d ' File search scope God good

| find a string in the form of or (or)

Egrep ' G (la|oo) d file Search range glad good

() + identification of multiple repeating groups

echo "Axyzxyzxyzxyzc" |egrep ' A (XYZ) C ' means a begins, C ends, and there is more than one "xyz" string in the middle

Note: In the regular expression is not a special character, if you want to check contains! With the < Word, grep ' [!>] ' file

[!a-z] So the response choice is wrong, [^a-z] this is correct.

Format Print: printf

Iii. use of the SED

SED is a non-interactive stream editor that can dynamically edit files. The object that SED handles is the data stream of the file. The working mode of SED is to compare each data row and perform a specific action if it conforms to the style.

The syntax of sed: sed ' style command ' file

This means that if a row of a file conforms to the ' style ', it executes the SED command specified, such as deleting (d) or substituting (s) or displaying (p).

The ' style ' here uses a pair//enclosed to indicate the meaning of the search. /1,6/first line to line 6th,/a/,/b/contains a to B rows

Note: SED does not change the original file content. SED works by reading the contents of the file, after the stream is edited, the results are displayed to the standard output, if you want to store the SED processing results, you should use the switch output to save the results of other files.

Options for SED:

Sed-n Use Quiet mode, in general sed mode, all data from stdin is generally listed on the screen, plus-n will only be listed after the SED-processed one line.

-E perform sed action editing directly in instruction mode

-F writes the SED action directly in a file, and-f filename performs the SED action within the filename.

-R SED Action support Enshin action notation

-I directly modifies the contents of the file read. Directly modify the contents of the original text, careful operation)

Action Description: [n1[,n2]]function

N1,N2 does not necessarily exist, generally represents the number of rows selected for the action, for example, if my action is required between 10 and 20 rows, then 10,20[action behavior]

Function has these things underneath:

A: new, a can be followed by a string, and these strings will appear on a new line (the next line in the current)

I: Insert, I can be followed by strings, and these strings will appear on a new line (the current line);

C: Replace, C can be followed by strings, these strings can replace the line between N1,N2!

D: Delete, because it is deleted ah, so d usually do not pick up any boom;

P: Printing, that is, printing a selected data. Normally p will work with parameter Sed-n ~

S: Replace, can directly be replaced by direct work! Usually this s action can be matched with regular notation! For example 1,20s/old/new/g is!

The syntax of sed:

Sed ' s/to replace the character/new character/g ' G denotes the global meaning

1. Delete a range of data rows

The sed ' 1,4d ' file deletes the 1th to 4th rows of data, leaving the remaining display. ($ stands for last line ' 1,$ ')

2. Delete the data row containing the style

The sed '/ab/d ' file removes all rows containing AB. The sed '/[0-9]\{3\}/d ' file removes the rows containing 3. \{3\} =//To look for a string of 3 numbers sed '/^$/d ' file to delete a blank line

3. Delete the line without style

The sed '/ab/!d ' does not contain AB's row deletions, and the rest is displayed. (or the line of the Han AB is not deleted

4. Display the data rows that contain the style (but SED also displays the rows of non-conforming data by default)

Sed '/ab/p ' shows the line containing AB, and the non-conforming also shows that the-n will suppress the default display of the other lines sed-n '/ab/p ' only shows the lines containing AB.

5. Replace

The sed-n ' s/ab/ab/p ' file converts the first occurrence of the AB string in each line of file to the Ab,sed-n ' s/ab/ab/gp ' file to change the AB of every line in file to AB

Sed-n ' s/ab//p ' file deletes the first AB in the file (the substitution of AB to an empty string is deleted).

The sed ' s/^...//' file removes the 3 characters from the beginning of each line, and if ^ is $, the last 3 characters of each line are deleted.

6. Take a string that matches the style

Sed-n ' s/\ (ab\)/\1cd/p ' file converts the first occurrence of the AB character in each line to ABCD. Can be decomposed into sed-n ' s/\ (ab\)/\1cd/p where () 1 is escaped with \, still is//mode

7. After finding the data that conforms to the style, replace the operation

Sed-n '/aaa/s/ab/ab/p ' file finds a row containing AAA, replace the first AB with AB (if it is GP, change the AB to the AB for each line appearing)

Sed-n '/AAA/,/BBB/S/123/456/GP ' file will contain AAA to BBB of those lines, all the 123 into 456

Sed-n ' 2,4s/a/b/p ' file from line 2nd to line 4th, replace 1th A to B

Example:

cat/etc/passwd| SED ' 2a Hello world '

After line 2nd, add a line that is HelloWorld, if there is no number before a. In a row behind each line, the content is Hello World (if the preceding words use i)

cat/etc/passwd| Sed ' 2i one \ n '

Add 2 rows to the front of line 2nd, respectively, one, two, separated by \ (carriage return) between each line

cat/etc/passwd | Sed ' 2i 1 \

2 \

4 {

3 {

5 '

Add 5 lines to the front of line 2nd, respectively 1 2 3 4 5

cat/etc/passwd | sed ' 1c Hello World ' to change the contents of line 1th to Hello World

cat/etc/passwd | sed ' 2,4c Hello World ' to change the contents of 第2-4 to Hello World

cat/etc/passwd | sed-n ' 1p ' shows line 1th if Sed-n ' 2,3p ' is displayed for lines 2nd and 3rd. Sed ' $p ' displays last line

grep-v ' ^$ ' file | grep-v ' ^# ' and egrep-v ' ^$|^# ' regular_express.txt $ represent the last line to remove blank lines and rows with the beginning of #

cat/etc/passwd |sed-e ' 1d '-e ' 2,3c Hello World ' delete line 1th and change the contents of line 2nd and 3rd to Hello World

Attention:

If SED is followed by multiple options, be sure to use the-e parameter

Iv. usage of awk

Awk is a language that can process data and produce formatted reports. It works by reading data, thinking each row of data as a record, dividing each record into fields with field separators, and then outputting the values for each field.

Awk applies a "style {action}" to each record, and executes the specified action if the row conforms to the style.

One of the styles or actions can be omitted. If there is only a style, indicates that you want to display rows that conform to the style, or if only the action is performed on each row of data.

Commonly used formats for awk:

awk "Style" file: Displays data rows that conform to the style.

awk ' {action} ' file: Performs the action in {} on each row.

The awk ' style {action} ' performs the action in {} on a data row that conforms to the style.

Several uses of awk:

1. awk '/root/' file

Displays the row of root in file

2. awk ' {print $1,$2} ' file "," can omit if two characters are separated by a space, use awk ' {print ' \ t '

Displays the 1th and 2nd fields of each row in file (the default is a space delimiter)

3. awk '/ab/{print $1,$2} ' file

Displays the 1th and 2nd fields of the row with AB in file (separated by a space)

4, Awk-f: '/^root/{print $2,$3} ' file

Use ":" As a delimiter to display the 2nd and 3rd characters in a line starting with root.

5, Awk-f: ' begin{ofs= "++++"}/^root/{print $1,$2,$3} '/etc/passwd

Put passwd, ":" As the delimiter to find the first line of behavior root, and display the first 3 fields, and the fields are separated by + +

Instance:

Variable names represent meaning

NF Total number of fields per row (default with a space or tab delimiter)

NR What awk is currently dealing with is the first few rows of data

FS Current delimiter, default is space key

[[email protected] ~]# cat 1a2a3a4a5ab2b3b4b5bc2c3c4c5c[[email protected] ~]# cat 1|awk ' {print $ \ t line: "NR" \ t the row total field: " NF} ' a line: 1 The total field of the row: 5b row: 2 The total field of the row: 5c row: 3 The Total field of the row: 5 Check the row of data that is less than 10 passwd the third column, and list only the account number and the third column. Cat/etc/passwd|awk ' {fs= ': '}$3<10 {print ' \ t ' $ ' root:x:0:0:root:/root:/bin/bashbin1daemon2...mail8

The first line above is all shown, because when we read the first row, the variables are "$" and "$". The default is still separated by the spacebar, so although we define the fs= ":", it only takes effect after the second line.

So what do we do? We can pre-set awk variables! Use the word BEGIN! Do this:Cat/etc/passwd|awk ' begin{fs= ': '}$3<10 {print ' \ t ' $ $} '

Show IP Address

Ifconfig |grep ' inet addr ' |grep Bcast|awk ' {print $} ' |awk-f: ' {print $} '

Show Network Name

Cat/proc/net/dev |awk-f: '/eth.:| sit.:| wlan.:| ppp.:/ {print '} '

Sed and awk can write a single book, which is very powerful, this article only describes the simplest usage.

These are reading notes, and then we really go into the shell combat!

This article is from the "Welcome to Linux World" blog, so be sure to keep this source http://linuxnote.blog.51cto.com/9876511/1641278

Linux Shell's ten sed and awk