Sed && awk tools and some common shell scripts

(i) SED

Sed is a streamlined, non-interactive, streaming editor that enters edit commands and specifies file names on the command line, and then looks at the output on the screen.

The content of a progressive read file is stored in a temporary buffer called pattern space, followed by the SED command to process the contents of the buffer, and the contents of the buffer are sent to the screen after processing is complete. Then the next line is processed, so it repeats until the end of the file. The original file love your content has not changed.

Sed ' 4, $d ' test.in # delete 4~ last line sed ' 3q ' test.in # read to the specified line and exit sed ' s/public/public/' test.in # Replace public for Publicsed-n ' s/public/ public/p ' test.in #打印匹配行

sed [options] sed_cmd files

-e Connect multiple edit commands
-f Specifies the SED script file name
-n block automatic output of input lines

Often grep can also achieve results, such as counting the number of rows in a specified file
count=$ (cat $FILE | grep-a "keyword" | wc-l)

Three ways to specify multiple commands:
1. Separating commands with semicolons

Sed ' s/public/public/;s/north/north/' test.in

2. Place the-e before each command

Sed-e ' s/public/public '-e ' S/north/north ' test.in

3. Using the branch command function, press ENTER after entering the single quotation mark to appear a multiline prompt (>)

Sed ' > s/public/public/> s/northnorth ' test.in

Common positioning commands:

Common editing commands:

P: Print matching line sed-n ' 3,5p ' test,in=: Displays the line number of the matching row sed-n '/north/= ' test.ind: Delete the matching row sed-n '/north/d ' test.ina\: Append one or more lines of text after the specified line, and display Add new content to the display. Sed '/NORTH/A\AAA>BBB>CCC ' test.ini\: Inserts one or more lines of text before the specified line, and displays the new content, as above. C\: Replaces the specified line with new text, using the format above L: Displays all characters in the specified line, including control characters (nonprinting characters) sed-n '/north/l ' test.ins: Replace command, using format as [address] S/OLD/NEW/[GPW] Address: If omitted, means edit all rows g: Global substitution P: Print the modified line W fname: Writes the replaced content to the specified file Sed-n ' s/north/north/gp ' test.insed-n ' s/north/north/ W data ' test.insed ' s/[0-9][0-9]$/&.5/' datafile& symbol is used in the replacement string to represent the substituted string R: Read the file, append the contents of another file to the specified line after sed ' $r data ' TEST.INW: Write a file, write the specified line to another file Sed-n '/public/w date2 ' Test.inn: reads the following line of the specified line into the edit buffer sed-n '/public/{n;s/north/north/p} ' test.in when using multiple sed edit commands on a specified line, enclose the braces, separating the commands with semicolons; Q: Exit, after reading to the specified line, exit sedsed ' 3q ' test.in

(ii) awk

Awk can work with columns
Simple usage:
awk [Options] sed_script files
-f Specifies the delimiter for the input record field, the value of the default environment variable IFS
-F reads from the specified file Awk_script
-V Sets the variable for awk

Awk-f: ' {print $} '/etc/passwdawk-f: '/root{print $ ' | ' $ '/etc/passwd

The execution process of awk:
1. If there is a Begin,awk first perform his assigned actions
2. Awk reads a line from the input, called a message record
3. awk divides the records that are read into several fields, puts the first field in the variable, the second in the variable, and so on, and then the whole record; The field delimiter can be specified by the option-F, otherwise the default delimiter is used.
4. Compare the current input records sequentially with the pattern in each awk_cmd:
If they match, the corresponding actions are executed
If it does not match, skip the corresponding actions until all awk_cmd are completed.
5. When an input record is complete, awk reads the next line of input and repeats the process until all the inputs have been processed.
6. When awk finishes processing all of the inputs, if there is an end, the appropriate actions are executed
7. If the input is a list of files, awk will process each file in the list sequentially

Examples of awk processing:

Ifconfig | awk '/inet addr/{print $} ' | Awk-f: ' {print-$} ' ifconfig | awk '/inet addr/{print $ ' |awk-f: ' Begin {print ' Begin ... '} ' {print $ {} end {print ' End ... '} '

In addition, awk can also handle compound expressions:

Finally, awk is not just a command, it's more of a programming language
Variable:
1. Internal variable awk ' {print nr,$0} ' #给文件加上行号
2. Custom variables
Function:
1. Built-in functions
2. Custom Functions
awk ' {print sum ($1,$2)} function sum (x, y) {S=x+y;return s} ' grade.txt
Array
awk ' BEGIN {print split ("123#456", arr, "#"); for (i in arr) {print arr[i]}} '

Because of these complexities, this article is no longer introduced.

finally I enclose a few simple shell scripts, more instances on my Github .

(1) Compare size

#!/bin/bashecho "Please input the numbers:" Read aread bif (($a = = $b)), then    echo "a = B" elif (($a > $b)) and then    E Cho "a > B" Else    echo "A < B" fi

(2) Find Files

#!/bin/bashecho "Enter a file name:" Read AIF [-e/home/tach/$a];then    echo "The file is exist!" else    echo "The file is not exist!" Fi

(3) Delete files of size 0

#!/bin/bashfor filename in ' ls ' does         a=$ (ls-l $filename | awk ' {print $} ')        if (a==0), then            rm $filename        F Idone

(4) View native IP address

#!/bin/bashifconfig | grep "inet Address:" | awk ' {print $} ' | Sed ' s/address: '//g

(5) IP address legality judgment

#!/bin/bashcheckipaddr () {   #IP地址需由三个. Delimited and all digital   echo $ | grep ^[0-9]\{1,3\}\.\ ([0-9]\{1,3\}\.\) \{2\}[0-9]\{ 1,3\}$ ">/dev/null   if [$?-ne 0];then       return 1   fi   ipaddr=$1   a= ' echo $ipaddr | awk-f. ' {print $ } '   b= ' echo $ipaddr | awk-f ' {print $} '   c= ' echo $ipaddr | awk-f. ' {print $} '   d= ' echo $ipaddr | awk-f. ' {print $4} ' for   num in $a $b $c $d do       if [$num-gt 255-o $num-lt 0];then           return 1       fi   D One   return 0}if [$#-ne 1];then    echo "Usage: $ ipaddr."    Exitelse    checkipaddr $    ans=$?    If [$ans-eq 0];then        echo "Legal IP address."    else        echo "Unlegal IP address."    Fifi

(6) Other

#!/bin/bash# Display current date and time echo ' date +%y-%m-%d-%h:%m:%s ' #查看哪个IP地址连接的最多netstat-an | grep established | awk ' {print $} ' |awk-f: ' {print '} ' | Sort | Uniq-c#awk do not sort delete duplicate rows awk '!x[$0]++ ' filename #x只是一个数据参数的名字, is a map that makes a specified logical judgment, and if the logical judgment is established, executes the specified command; No, just skip this line. See the 10 UNIX commands most commonly used awk ' {print $ ' ~/.bash_history | Sort | uniq-c | Sort-rn | The-R in Head-n 10#sort is descending, _-n is sorted by value (default comparison character, 10<2) #逆序查看文件cat 1.txt | awk ' {a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--]} ' #查看第3到6行awk ' Nr >=3 && nr <=6 ' filename #crontab文 Some examples of pieces * * * */usr/local/etc/rc.d/lighttpd restart #每晚9.30 Restart Apache45 4 1,10,22 */usr/local/etc/rc.d/lighttpd Restar T #每月的1, 10, 22nd 4:45 Restart apache10 1 * * 6,0/usr/local/etc/rc.d/lighttpd restart #每周六, day of 1:10 restart apache0,30 18-23 * * */usr/l OCAL/ETC/RC.D/LIGHTTPD Restart #表示在每天18.00 to 23.00 restarts every 30 minutes apache* 23-7/1 * * */usr/local/etc/rc.d/lighttpd Restart # From 11 o'clock to 7 in the morning, restart Apache every hour.

Sed && awk tools and some common shell scripts