Awk Implementation of SED single-line script Quick Reference

Http://linuxtoy.org/archives/sed-awk.html

{Author/bones7456}

Sed and awk are common stream editors in Linux. They have their own unique characteristics. This article does not compare them, but is about to use them as fun. Let's take the SED single-line script quick reference, done again with awk ~ It's really hard to comment on it. Generally, sed commands are shorter and harder to understand. awk is longer, but if and while commands are more logical and more like "programs ". Which one do you like to use? Let the readers decide on their own!

Text interval:

# Add an empty row after each row

sed Gawk '{printf("%s\n\n",$0)}'

# Delete all original empty rows and add an empty row after each row.
# In this way, each row in the output text is followed by an empty row.

sed '/^$/d;G'awk '!/^$/{printf("%s\n\n",$0)}'

# Add two blank rows after each row

sed 'G;G'awk '{printf("%s\n\n\n",$0)}'

# Delete all empty rows generated by the first script (that is, delete all even rows)

sed 'n;d'awk '{f=!f;if(f)print $0}'

# Insert an empty row before matching the row with the style "RegEx"

sed '/regex/{x;p;x;}'awk '{if(/regex/)printf("\n%s\n",$0);else print $0}'

# Insert an empty row after matching the row with the style "RegEx"

sed '/regex/G'awk '{if(/regex/)printf("%s\n\n",$0);else print $0}'

# Insert an empty row before and after the row matching the style "RegEx"

sed '/regex/{x;p;x;G;}'awk '{if(/regex/)printf("\n%s\n\n",$0);else print $0}'

No:

# Number each row in the file (simple left alignment ). The "tab" is used here"
# (Tab, see the '\ t' usage description at the end of this article) instead of spaces to align edges.

sed = filename | sed 'N;s/\n/\t/'awk '{i++;printf("%d\t%s\n",i,$0)}'

# Number of all rows in the file (the row number is left and the right of the text is aligned ).

sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'awk '{i++;printf("%6d  %s\n",i,$0)}'

# Number of all rows in the file, but only the row numbers of non-blank rows are displayed.

sed '/./=' filename | sed '/./N; s/\n/ /'awk '{i++;if(!/^$/)printf("%d %s\n",i,$0);else print}'

# Calculate the number of rows (simulate "WC-L ")

sed -n '$='awk '{i++}END{print i}'

Text conversion and substitution:

# Unix environment: the new line character (CR/LF) for converting DOS is in UNIX format.

Sed's /. $ // '# assume that all rows end with Cr/lf sed's/^ m $ //' # In bash/tcsh, change Ctrl-m to Ctrl-vsed's/\ x0d $ // '# ssed, gsed 3.02.80, and higher awk' {sub (/\ x0d $ /, ""); print $0 }'

# Unix environment: Convert the Unix newline character (LF) to the DOS format.

Sed "s/$/'echo-e \ R'/" # The command SED's/$ 'Used in KSh "/'echo \ R' /"# command sed used in Bash" s/$/'echo \ R'/"# command SED's/$/\ used in zsh /\ r/'# gsed 3.02.80 and later awk' {printf ("% s \ r \ n ", $0 )}'

# DOS environment: Convert UNIX newline character (LF) to DOS format.

Sed "s/$ //" # method 1sed-n p # method 2

DOS environment skipped

# DOS environment: Convert the DOS newline character (CR/LF) to Unix format.
# The following script is only valid for unxutils sed 4.0.7 and later versions. To identify the unxutils version
# Sed can use its unique "-text" option. You can use the help option ("-help") to view
# Whether there is a "-text" item to determine whether the version is unxutils. Other DoS
# The SED version cannot perform this conversion. However, you can use "TR" to achieve this conversion.

Sed "s/\ r //" infile> OUTFILE # unxutils sed v4.0.7 or later tr-d \ r <infile> OUTFILE # GNU tr 1.22 or later

DOS environment skipped

# Delete the leading blank characters (spaces and tabs) of each line
# Align left

Sed's/^ [\ t] * // # For the description of '\ t' usage at the end of this article, awk' {sub (/^ [\ t] + /, ""); print $0 }'

# Delete the blank characters (spaces and tabs) at the end of each line

Sed's/[\ t] * $ // # For the description of '\ t' usage at the end of this article, awk' {sub (/[\ t] + $ /, ""); print $0 }'

# Delete leading and trailing blank characters in each row

sed 's/^[ \t]*//;s/[ \t]*$//'awk '{sub(/^[ \t]+/,"");sub(/[ \t]+$/,"");print $0}'

# Insert 5 spaces at the beginning of each line (to move the full text to the right five characters)

sed 's/^/     /'awk '{printf("     %s\n",$0)}'

# Align all texts right with 79 characters in width
#78 characters plus the last space

sed -e :a -e 's/^.\{1,78\}$/ &/;ta'awk '{printf("%79s\n",$0)}'

# Use 79 characters as the width to center all texts. In method 1, to center the text before each row
# The header and the backend header are filled with spaces. In method 2, the text is only filled before the text in the center process.
# Spaces, and half of these spaces will be deleted. In addition, no spaces are filled in the backend of each row.

Sed-E: A-E's/^. \ {\} $/&/; TA '# method 1sed-E: A-E's/^. \ {\} $/&/; TA '-E's/\ (* \) \ 1/\ 1/' # method 2awk '{for (I = 0; I <39-length ($0)/2; I ++) printf (""); printf ("% s \ n", $0 )} '# equivalent to method 2 above

# Search for the string "foo" in each row and replace the "foo" with "bar"

Sed's/Foo/BAR/'# Replace only the first "foo" string SED's/Foo/BAR/4' in each row # Replace only the fourth "in each row foo "string SED's/Foo/BAR/G' # Replace all" foo "in each row with" bar "SED's /\(. * \) Foo \(. * Foo \)/\ 1bar \ 2/'# Replace the last and second "foo" SED's /\(. * \) Foo/\ 1bar/'# Replace the last "foo" awk' {gsub (/Foo/, "bar "); print $0} '# Replace all "foo" in each row with "bar"

# Replace "foo" with "bar" only when the string "Baz" appears in the row"

sed '/baz/s/foo/bar/g'awk '{if(/baz/)gsub(/foo/,"bar");print $0}'

# Replace "foo" with "bar", and replace it only when "Baz" is not displayed in the row.

sed '/baz/!s/foo/bar/g'awk '{if(/baz$/)gsub(/foo/,"bar");print $0}'

# "Red" is used for both "Scarlet", "Ruby", and "puce"

Sed's/scarlet/red/g; S/Ruby/red/g; s/puce/red/G' # gsed's/scarlet \ | Ruby \ | puce/red/G' # awk only for GNU sed '{ gsub (/scarlet | Ruby | puce /, "Red"); print $0 }'

# Invert all rows. The first line is the last line, and so on (simulate "TAC ").
# For some reason, hhsed v1.5 deletes empty lines in the file when the following command is used

Sed '1! G; h; $! D' # method 1sed-n' 1! G; h; $ P' # method 2awk '{A [I ++] = $0} end {for (j = I-1; j> = 0; j --) print a [J]}'

# Sort the characters in the row in reverse order. The first word becomes the last word ,...... (Simulate "Rev ")

sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'awk '{for(i=length($0);i>0;i--)printf("%s",substr($0,i,1));printf("\n")}'

# Concatenate each two rows into one line (similar to "Paste ")

sed '$!N;s/\n/ /'awk '{f=!f;if(f)printf("%s",$0);else printf(" %s\n",$0)}'

# If the current row ends with a backslash (\), the next row is added to the end of the current row.
# Remove the backslash at the end of the original line

sed -e :a -e '/\\$/N; s/\\\n//; ta'awk '{if(/\\$/)printf("%s",substr($0,0,length($0)-1));else printf("%s\n",$0)}'

# If the current row starts with an equal sign, add the current row to the end of the previous row
# Replace the "=" of the original line header with a single space"

sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'awk '{if(/^=/)printf(" %s",substr($0,2));else printf("%s%s",a,$0);a="\n"}END{printf("\n")}'

# Add a comma separator for the numeric string and change "1234567" to "1,234,567"

Gsed ': A; S/\ B [0-9] \ {3 \} \>/, &/; TA' # GNU sedsed-E: a-E's /\(. * [0-9] \) \ ([0-9] \ {3 \} \)/\ 1, \ 2/; TA '# other SED

# Awk regular expressions are not followed by backward matching and reference, which is quite embarrassing.

awk '{while(match($0,/[0-9][0-9][0-9][0-9]+/)){$0=sprintf("%s,%s",substr($0,0,RSTART+RLENGTH-4),substr($0,RSTART+RLENGTH-3))}print $0}'

# Add a comma separator (gnu sed) for values with decimal points and negative numbers)

gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'

# Similar to the previous example

awk '{while(match($0,/[^\.0-9][0-9][0-9][0-9][0-9]+/)){$0=sprintf("%s,%s",substr($0,0,RSTART+RLENGTH-4),substr($0,RSTART+RLENGTH-3))}print $0}'

# Add a blank row after each 5 rows (add a blank row after rows 5, 10, 15, 20, and so on)

Gsed '0 ~ 5G '# only applies to GNU sed: 'n'; n; G;' # other sedawk '{print $0; I ++; if (I = 5) {printf ("\ n"); I = 0 }}'

Select to display specific rows:

# Display the first 10 lines in the file (simulate the "head" behavior)

sed 10qawk '{print;if(NR==10)exit}'

# Display the first line of the file (simulate the "head-1" command)

sed qawk '{print;exit}'

# Display the last 10 lines in the file (simulate "tail ")

sed -e :a -e '$q;N;11,$D;ba'

# Using awk to do this is a little deficient and requires full-text caching. It must be slow for large files.

awk '{A[NR]=$0}END{for(i=NR-9;i<=NR;i++)print A[i]}'

# Display the last two lines in the file (simulate the "tail-2" command)

sed '$!N;$!D'awk '{A[NR]=$0}END{for(i=NR-1;i<=NR;i++)print A[i]}'

# Display the last line in the file (simulate "tail-1 ")

Sed '$! D' # method 1sed-n' $ P' # method 2

# This is easy to do. Only the last row is saved.

awk '{A=$0}END{print A}'

# Display the second and last lines in the file

Sed-e '$! {H; D;} '-e x # when there is only one row in the file, the output empty line sed-e '1 {$ q;}'-e' $! {H; D;} '-e x # when there is only one row in the file, the sed-e '1 {$ D;}'-e' $! {H; D;} '-e x # when there is only one row in the file, no output

# Store two rows (empty rows are output when there is only one row in the file)

awk '{B=A;A=$0}END{print B}'

# Only display rows matching Regular Expressions (simulate "grep ")

Sed-n'/Regexp/P' # method 1sed '/Regexp /! D' # method 2awk '/Regexp/{print }'

# Show only the rows that do not match the regular expression (simulate "grep-V ")

Sed-n'/Regexp /! P' # method 1, which corresponds to SED '/Regexp/d' # method 2, similar to the syntax awk '! /Regexp/{print }'

# Search for "Regexp" and display the last line of the matched row, but not the matched row

sed -n '/regexp/{g;1!p;};h'awk '/regexp/{print A}{A=$0}'

# Search for "Regexp" and display the next row of the matching row, but not the matching row

sed -n '/regexp/{n;p;}'awk '{if(A)print;A=0}/regexp/{A=1}'

# Display the rows that contain "Regexp" and the rows before and after it, and add "Regexp" to the row's line number (similar to "grep-A1-B1") before the first line ")

sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e hawk '{if(F)print;F=0}/regexp/{print NR;print b;print;F=1}{b=$0}'

# Display rows containing "AAA", "BBB", and "CCC" (in any order)

Sed '/AAA /! D;/BBB /! D;/CCC /! D' # The string order does not affect the result awk '{If (MATCH ($0,/AAA/) & match ($0,/BBB /) & match ($0,/CCC/) print }'

# Display rows containing "AAA", "BBB", and "CCC" (fixed order)

sed '/AAA.*BBB.*CCC/!d'awk '{if(match($0,/AAA.*BBB.*CCC/))print}'

# Display rows that contain "AAA" "BBB" or "CCC" (simulate "egrep ")

Sed-e '/AAA/B'-e '/BBB/B'-e '/CCC/B'-e d # most sedgsed '/AAA \ | BBB \ | ccc /! D' # awk for GNU sed '/AAA/{print; next}/BBB/{print; next}/CCC/{print} 'awk'/AAA | BBB | CCC/{print }'

# Display the section containing "AAA" (separated by blank lines)
# Hhsed v1.5 must add "g;" after "X;". This is the case for the next three scripts.

sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'awk 'BEGIN{RS=""}/AAA/{print}'awk -vRS= '/AAA/{print}'

# Display paragraphs containing "AAA", "BBB", and "CCC" strings (in any order)

sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'awk -vRS= '{if(match($0,/AAA/) && match($0,/BBB/) && match($0,/CCC/))print}'

# Display the section containing any string of "AAA", "BBB", and "CCC" (in any order)

Sed-e '/./{H; $! D;} '-e' X;/AAA/B'-E'/BBB/B '-E'/CCC/B'-e dgsed '/. /{H; $! D ;}; X;/AAA \ | BBB \ | CCC/B; d' # valid only for GNU sed awk-VRS = '/AAA | BBB | CCC/{print ""; print }'

# Display rows containing 65 or more characters

sed -n '/^.\{65\}/p'

Cat ll.txt | awk '{If (length ($0)> = 65) print }'

# Display rows with less than 65 characters

Sed-n'/^. \ {65 \}/! P' # method 1, which corresponds to SED '/^. \ {65 \}/d' # method 2, simpler method awk '{If (length ($0) <= 65) print }'

# Display part of the text-starting from the row containing the regular expression to the end of the last row

sed -n '/regexp/,$p'awk '/regexp/{F=1}{if(F)print}'

# Display part of text -- specify the row number range (from 8th to 12th rows, including 8 and 12 rows)

Sed-n'8, 12p' # method 1sed '8, 12! D' # method 2awk '{If (NR> = 8 & Nr <12) print }'

# Display rows 52nd

Sed-N '52p' # method 1sed '52! D' # method 2sed '52q; D' # method 3: awk is more efficient when processing large files '{If (Nr = 52) {print; exit }}'

# Display each 7 rows starting from 3rd

Gsed-n' 3 ~ 7p' # only applies to GNU sed-n' 3, $ {P; n ;} '# other sedawk' {If (Nr = 3) F = 1} {If (f) {I ++; if (I % 7 = 1) print }}'

# Display the text (inclusive) between two regular expressions)

Sed-n'/Iowa/,/Montana/P' # case sensitive awk '/Iowa/{f = 1} {If (f) print}/Montana/{f = 0 }'

Select to delete a specific row:

# Display the entire document except the content between two regular expressions

sed '/Iowa/,/Montana/d'awk '/Iowa/{F=1}{if(!F)print}/Montana/{F=0}'

# Delete adjacent duplicate rows in the file (simulate "uniq ")
# Only the first row in the duplicate row is retained, and the other rows are deleted.

sed '$!N; /^\(.*\)\n\1$/!P; D'awk '{if($0!=B)print;B=$0}'

# Delete duplicate rows in the file, regardless of whether there are adjacent rows. Note the cache size supported by hold space, or use GNU sed.

Sed-n'g; S/\ n/&/;/^ \ ([-~] * \ N \). * \ n \ 1/d; S/\ N //; h; P' # bones7456 Note: I cannot run this command here. awk '{If (! ($0 in B) print; B [$0] = 1 }'

# Delete all rows except duplicate rows (simulate "uniq-d ")

sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'awk '{if($0==B && $0!=l){print;l=$0}B=$0}'

# Delete the first 10 lines in the file

sed '1,10d'awk '{if(NR>10)print}'

# Delete the last row in the file

sed '$d'

# Awk does not know the total number of lines of files in the process, so it can only be cached throughout the article. Large files may not be suitable. The two below are also the same

awk '{B[NR]=$0}END{for(i=0;i<=NR-1;i++)print B[i]}'

# Delete the last two lines in the file

sed 'N;$!P;$!D;$d'awk '{B[NR]=$0}END{for(i=0;i<=NR-2;i++)print B[i]}'

# Delete the last 10 lines in the file

Sed-E: A-E '$ D; n; 2, 10ba'-e 'P; D' # method 1sed-n-e: A-E '1, 10! {P; n; D ;}; N; Ba' # method 2awk '{B [Nr] =0 0} end {for (I = 0; I <= NR-10; I ++) print B [I]}'

# Delete multiple rows of 8

Gsed '0 ~ 8d '# only valid for GNU sed 'n'; n; D;' # other sedawk '{If (NR % 8! = 0) print} '| HEAD

# Deleting matching rows

Sed '/pattern/d' # deletes rows containing pattern. Of course, pattern can be replaced with any valid Regular Expression awk '{If (! Match ($0,/pattern/) print }'

# Delete all empty lines in the file (same effect as "grep)

Sed '/^ $/d' # method 1sed '/./! D' # method 2awk '{If (! Match ($0,/^ $/) print }'

# Only the first row of multiple adjacent empty rows is retained. Delete the empty lines at the top and end of the file.
# (Simulate "cat-S ")

Sed '/./,/^ $ /! D' # Method 1: Delete the empty row at the top of the file and allow the end to keep an empty row sed '/^ $/N; // \ N $/d' # method 2, allow an empty row to be retained at the top, and leave no blank lines at the end of awk '{If (! Match ($0,/^ $/) {print; F = 1} else {If (f) print; F = 0} '# method 2 above

# Only the first two rows of multiple adjacent empty rows are retained.

sed '/^$/N;/\n$/N;//D'awk '{if(!match($0,/^$/)){print;F=0}else{if(F<2)print;F++}}'

# Delete all blank lines at the top of the file

sed '/./,$!d'awk '{if(F || !match($0,/^$/)){print;F=1}}'

# Delete all empty lines at the end of the file

Sed-E: A-E '/^ \ n * $/{$ D; n; BA'-e'} '# Use sed-E for all sed: a-E '/^ \ n * $/N;/\ N $/Ba' # Same as above, but only for gsed 3. 02. * Valid awk '/^. + $/{for (I = L; I <NR-1; I ++) print ""; print; L = nR }'

# Delete the last line of each paragraph

sed -n '/^$/{p;h;};/./{x;/./p;}'

# It's very long and uugly. There should be a better way

awk -vRS= '{B=$0;l=0;f=1;while(match(B,/\n/)>0){print substr(B,l,RSTART-l-f);l=RSTART;sub(/\n/,"",B);f=0};print ""}'

Special applications:

# Remove the nroff mark from the man page. In UNIX System V or bash shell
# You may need to add the-e option when using the 'echo 'command.

Sed "s /. 'echo \ B '// G "# The outer double brackets are required (UNIX environment) SED's /. ^ h // G' # In bash or tcsh, press Ctrl-V and then Ctrl-hsed's /. \ x08 // G' # sed 1.5, gnu sed, and ssed use the hexadecimal representation awk '{gsub (/. \ x08/, "", $0); print }'

# Extract the header of a news group or email

Sed '/^ $/Q' # delete all content after the first empty line awk' {print}/^ $/{exit }'

# Extract the body of a newsgroup or email

Sed '1,/^ $/d' # delete all content before the first empty line awk '{If (f) print}/^ $/{f = 1 }'

# Extract "subject" (title bar field) from the mail header and remove the "Subject:" at the beginning

sed '/^Subject: */!d; s///;q'awk '/^Subject:.*/{print substr($0,10)}/^$/{exit}'

# Obtain the reply address from the email header

sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

# It seems that the first line starting with reply-to is output? Why is from used? The rules are unclear ..

awk '/^Reply-To:.*/{print;exit}/^$/{exit}'

# Obtain the email address. Based on the line of mail headers generated by the previous script, the non-email addresses are further removed. (See the previous script)

sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

# What is in angle brackets?

awk -F'[<>]+' '{print $2}'

# Add angle brackets and spaces (reference information) at the beginning of each line)

sed 's/^/> /'awk '{print "> " $0}'

# Delete the angle brackets and spaces at the beginning of each line (unreference)

sed 's/^> //'awk '/^> /{print substr($0,3)}'

# Remove most HTML tags (including cross-row tags)

sed -e :a -e 's/<[^>]*>//g;/</N;//ba'awk '{gsub(/<[^>]*>/,"",$0);print}'

# Decode uencode files that are divided into multiple volumes. Removes the file header information and only keeps the uencode part.
# The file must be transmitted to sed in a specific order. The script of the first version can be directly input in the command line;
# The second version can be placed in a shell script with the execution permission. (One of Rahul dhesi
# The script is modified .)

sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode   # vers. 1sed '/^end/,/^begin/d' "$@" | uudecode                    # vers. 2

# I don't want to install udecode verification. Write it roughly.

awk '/^end/{F=0}{if(F)print}/^begin/{F=1}' file1 file2 ... fileX

# Sort the paragraphs in the file alphabetically. Paragraphs are separated by (one or more) blank rows. Use GNU SED
# The character "\ v" is used to represent vertical tabs. It is used as a placeholder for line breaks-you can also
# Replace it with other characters that are not used in the file.

sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'awk -vRS= '{gsub(/\n/,"\v",$0);print}' ll.txt | sort | awk '{gsub(/\v/,"\n",$0);print;print ""}'

# Compress each. txt file, compress the original file, and delete the compressed. ZIP file.
# Name it the same as the original name (only with Different Extensions ). (DOS environment: "DIR/B"
# Display file names without paths ).

echo @echo off >zipup.batdir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

The DOS environment is skipped again, and I think it is more handsome to replace the bash parameter paiiyun.txt0000.zip here.

Some sed descriptions below are skipped. You need to view the original text by yourself.

{Source. Thanks bones7456 .}