Sed & amp; awk tool and some common shell scripts

Sed & awk tools and some common shell scripts

(1) sed

Sed is a streamlined and non-interactive stream editor. It enters the edit command in the command line and specifies the file name, and then displays the output on the screen.

The file content read row by row is stored in the temporary buffer, which is called the pattern space. Then, the sed command is used to process the buffer content, send the buffer content to the screen. Next, process the next row, and repeat until the end of the file. The content of the original file that loves you has not changed.

 

Sed '4, $ d' test. in # Delete 4 ~ The last row of sed '3q' test. in # exit sed's/public/PUBLIC/'test after reading the specified row. in # replace public with PUBLICsed-n's/public/PUBLIC/P' test. in # print matching rows

Sed [options] sed_cmd files

 

-E connects multiple editing commands
-F specifies the sed script file name
-N: prevents automatic output of input lines

Generally, grep can achieve the same effect, for example, counting the number of rows in a specified file.
COUNT = $ (cat $ FILE | grep-a "keyword" | wc-l)

Three methods for specifying multiple commands:
1. Use semicolons to separate commands

sed 's/public/PUBLIC/;s/north/NORTH/' test.in

2. Place-e before each command

sed -e 's/public/PUBLIC' -e 's/north/NORTH' test.in

3. Use the branch command function. Press Enter after entering single quotes, multiple lines of prompt (>) will appear)

sed '> s/public/PUBLIC/> s/northNORTH' test.in

 

Common positioning commands:

Common editing commands:

P: print the matched row sed-n '3, 5' test, in =: display the row number sed-n'/north/= 'test. ind: Delete the matched row sed-n'/north/d' test. ina \: append one or more lines of text after a specified row and display the added content. Sed '/north/a \ AAA \> BBB \> CCC' test. ini \: insert one or more lines of text before the specified line and display the new content. C \: Replace the specified row with new text. Use the same format as above l: Display All characters in the specified row, including control characters (non-print characters) sed-n'/north/l' test. ins: replace command, in the format of [address] s/old/new/[gpw] address: If omitted, it indicates editing all rows g: replace Global p: print the modified row w fname: Write the replaced content to the specified file sed-n's/north/NORTH/gp 'test. insed-n's/north/NORTH/w data 'test. insed's/[0-9] [0-9] $ /&. 5/When 'datafile & 'is used to replace the string, it indicates the replaced string r: Read the file, and append the content of another file to the specified row and then sed' $ r data' test. inw: writes a specified row to the sed-n '/public/w date2' test file. inn : Read the following row of the specified row into the editing buffer sed-n '/public/{n; s/north/NORTH/p}' test. when you use multiple sed edit commands for a specified row, in must be enclosed in braces. commands are separated by semicolons. Q: exit. After reading the specified row, exit sedsed '3q' test. in.

(2) awk

Awk can process Columns
Simple usage:
Awk [options] sed_script files
-F specifies the delimiter of the input record field. The default value of the environment variable IFS
-F Read awk_script from the specified file
-V: Set variables for awk

awk -F:'{print $1}' /etc/passwdawk -F:'/root{print $1 "|" $3}' /etc/passwd

 

Awk execution process:
1. If BEGIN exists, awk first executes the specified actions
2. awk reads a row from the input, which is called a message record.
3. awk splits the read records into several fields, puts the first field in the variable $1, the second in the variable $2, and so on; $0 indicates the entire record; the field separator can be specified by option-F. Otherwise, the default Delimiter is used.
4. Compare the current input records with the pattern in each awk_cmd in sequence:
If they match, execute the corresponding actions
If they do not match, the corresponding actions will be skipped until all awk_cmd tasks are completed.
5. After an input record is completed, awk reads the next line of the input record and repeats the processing process until all the input is processed.
6. If END exists after awk finishes processing all input, execute the corresponding actions
7. If the input is a file list, awk will process each file in the list in order

Example of awk processing:

ifconfig | awk '/inet addr/{ print $2 }'| awk -F:'{ print $2 }'ifconfig | awk '/inet addr/{ print $2 }'|awk -F: 'BEGIN { print "begin..."}'{ print $2 } END { print "end..." }'

In addition, awk can also process compound expressions:

 

Finally, awk is not only a command, but also a programming language.
Variable:
1. The internal variable awk '{print NR, $0}' # adds a row number to the file.
2. Custom Variables
Function:
1. built-in functions
2. User-Defined Functions
Awk '{print sum ($1, $2)} function sum (x, y) {s = x + y; return s}' grade.txt
Array
Awk 'in in {print split ("123 #456", arr, "#"); for (I in arr) {print arr [I]}'

Because of these complexity, this article will not be discussed.

AttachedA few simple shell scripts, more instances in myGithub.

(1) Compare the size

#!/bin/bashecho "please input the two numbers:"read aread bif (($a==$b));then    echo "a = b"elif (($a>$b));then    echo "a > b"else    echo "a < b"fi

(2) Searching for files

#!/bin/bashecho "Enter a file name :"read aif [ -e /home/tach/$a ];then    echo "the file is exist!"else    echo "the file is not exist!"fi

(3) Delete an object 0 in size

#!/bin/bashfor filename in `ls`do         a=$(ls -l $filename | awk '{print $5}')        if ((a==0));then            rm $filename        fidone

(4) view the local IP Address

#! /Bin/bashifconfig | grep "inet address:" | awk '{print $2}' | sed's/address: '// g

(5) Determination of IP address Validity

#! /Bin/bashCheckIPAddr () {# Three IP addresses are required. separated by numbers echo $1 | grep "^ [0-9] \ {1, 3 \}\. \ ([0-9] \ {1, 3 \}\. \) \ {2 \} [0-9] \ {1, 3 \} $ ">/dev/null if [$? -Ne 0]; then return 1 fi ipaddr = $1 a = 'echo $ ipaddr | awk-F. '{print $1}' 'B = 'echo $ ipaddr | awk-F. '{print $2}' c = 'echo $ ipaddr | awk-F. '{print $3} 'd = 'echo $ ipaddr | awk-F. '{print $4} ''for num in $ a $ B $ c $ d do if [$ num-gt 255-o $ num-lt 0]; then return 1 fi done return 0} if [$ #-ne 1]; then echo "Usage: $0 ipaddr. "exitelse CheckIPAddr $1 ans =$? If [$ ans-eq 0]; then echo "legal ip address." else echo "unlegal ip address." fifi

(6) Others

#! /Bin/bash # display current date and time echo 'date + % Y-% m-% d-% H: % M: % s' # maximum netstat-an | grep ESTABLISHED | awk '{print $5}' | awk-F: '{print $1}' | sort | uniq-c # awk unordered deletion of duplicate rows awk '! X [$0] ++ 'filename# x is only the name of a Data parameter. It is a map and is used for the specified logic judgment. If the logic judgment is true, the specified command is executed; not true. Skip this line directly # view the 10 most commonly used unix Commands awk '{print $1 }'~ /. Bash_history | sort | uniq-c | sort-rn | head-n 10 # In sort,-r is in descending order, and _-n is in numerical order (default comparison character, 10 <2) # view the file cat 1.txt in reverse order | awk '{a [I ++] = $0} END {for (j = I-1; j> = 0 ;) print a [j --]} '# view lines 3rd to 6 awk 'nr> = 3 & NR <= 6' filename # example 30 21 ***/ usr/local/etc/rc. d/lighttpd restart # restart apache45 4, 10, 22, 9.30/usr/local/etc/rc every night. d/lighttpd restart # restart apache10 1*6, 0/usr/local/etc/rc at and every month. d/lighttpd restart # restart apache0, 30 18-23 ***/usr/local/etc/rc every Saturday and. d/lighttpd restart # indicates restarting apache * 23-18.00 ***/usr/local/etc/rc every 30 minutes between 23.00 and 7/1 every day. d/lighttpd restart # restart apache every one hour from PM to AM