Sed and awk in Linux

Linux Shell programming-awk usage
1. Use of awk
Basic functions: browses and extracts information based on specified rules in a file or string. Only after awk extracts information can other text operations be performed. A complete awk script is usually used to format text file information.
Call method:
1. Command Line
2. Insert all the awk commands into a file, make the awk program executable, and then use the awk command interpreter as the first line of the script to call it by typing the Script Name.
3. Insert all the awk commands into a separate file and call it.
Option Description:-F domain symbols are space characters by default.
-F indicates the awk script
2. modes and actions
1. All awk statements are composed of modes and actions. There can be many statements in an awk script.
The Mode part determines when the Action Statement is triggered and the event is triggered. The default mode is the execution status.
Processing is the operation on data.
2. The mode can be any conditional statement, compound statement, or regular expression.
3. The mode includes two special fields: begin and end.
Domains and records
Domain ID: $1, $2 ,..., $ N. Use commas to separate fields. $0 indicates all domains.
Print domain or all domain: Print command
Note: When an awk error occurs, you can find it:
1. Make sure that the entire awk command is enclosed in single quotes
2. Make sure all quotation marks in the command are enclosed in pairs.
3. Make sure that the Action Statement is enclosed in curly brackets and the Condition Statement is enclosed in parentheses.
4. Do not forget to use curly brackets
Conditional Operators
~ Match Regular Expression
!~ Do not match Regular Expression
Built-in variables:
NF: Number of domain names in each record
Is to pass the return value of the variable $ PWD into awk and display its directory.
You can use NF to obtain the file name.
Note: The field separator is/
Nr: number of records

1) Save the awk output: # awk '{print $0}' myfile> newfile
2). Use tee to output to the screen while outputting to the file (using pipeline '| ')
# Awk '{print $0}' myfile | tee newfile
3) print the report header, # awk 'in in {print "Hello, this is Title \ n ---------"} {print $0} 'newfile
4). Print the end of the message. # awk 'in in {print $0} end {"End of file."} 'myfile
5). Match. If the second column of myfile contains Brown, print the information as follows.
# Awk '{if ($2 ~ /Brown/) Print $0} 'myfile
6). Exact match. Same as above, only ~ Change to =
7). do not match. Sometimes you need to browse the information and extract records of the unmatched operations ~ The opposite symbol is !, As follows:
# Awk '$0 !~ /Brown/'myfile
8). Comparison
For example, less than: # awk 'if ($1 <$2) Print $1' myfile
9). Set case sensitivity # awk '[Gg] reen/'myfile
10). Any character, expression/^... A/indicates that the first three characters are any characters
# Awk '$1 ~ /^... A/'myfile
11). Or link match, # awk '$0 ~ /(Yellow | red)/'myfile
12). And: &, or: |
13). awk built-in Variables
<1>. to quickly view the number of records, use NR, for example: # awk 'end {print Nr} 'myfile <2>. use the NF variable to display the number of fields in each read record and print the input file name in the end part. # Awk '{print NR, NF, $0} end {print filename}' myfile
<3>. Determine and output. # awk '{If (NR> 0 & $4 ~ /Brown/) Print $0 'myfile
14). You can set the input domain to the domain variable name.
15). Compare the domain values.
@ 1. assign a value to the variable name in begin. Generally, it is helpful to assign values in the begin part, which can be greatly reduced when the awk expression is modified.
@ 2. Use the actual value in the Link operation and enclose it in parentheses when using the link operation.
# Awk '{if ($6 <27) Print $0}' myfile
# Awk 'in in {Baseline = "27"} {if ($6 <baseline) Print $6} 'myfile
17). Modify the data field value
When modifying any domain in the awk, it is important that the actual file can be modified, but the awk copy in the cache is changed, awk will reflect the Modification Trace in the NR or NF variable.
# Awk '{if ($1 = "M. tans") $6 = $6-1; print $6}' myfile
18). Modify text fields
# Awk '{if ($1 = "J. troll") ($1 = "J. L. troll"); print $1}' myfile
19). Only display modification records
# Awk '{if ($1 = "J. troll") {$1 = "J. L. troll"; print $1} 'myfile
20). Create a new output domain
When processing data in awk, it is a good habit to create a new domain based on each domain. Example:
# Awk 'in in {print "name Score \ t"} if ($6 <$7) {$8 = $7-$6; print $8} 'myfile
21). Add a column value.
# Awk 'tot + = $6; end {print "club total point." tot} 'myfile
22). Print the file name and its length in the mode and put it in the tot variable.
# Ls-L | awk '/^ [^ d] {print $9 "\ t" $5} {tot + = $5} end {print "Total KB: TOT "}'
23). awk built-in string functions
24). awk script file

As described above, the command is only placed in a file, and the file must be! /Bin/awk-F, because this will make this file self-explanatory. Otherwise, it will not work. For ease of separation, it is best to include a file extension. awk. After writing the file, use chmod U + X to make the file executable.

//////////////////////////////////////// //////////////////////////////////////// //////////////////////////////////////// //////////////////////////////////////// //////////////////////////////////////// ///////////////

The above are just some of my experiments. The awk command is very powerful. You can refer to some other manuals to get a clearer explanation.

Usage of sed in Linux Shell Programming
1. It is a non-interactive text stream editor. It edits a file or copies the text exported from standard input.
2. Specify the text line to be changed using the row number or regular expression.
3. Sed does not deal with the initial file, but only one copy of it. If the operation result is not redirected to a file, it will be output to the screen.
Call method:
A. Command Line
B. Insert the SED command into the script file, and then call SED
C. Insert the SED command into the script file so that the script file is executable.
Save sed output
Redirect to a new file>
How to locate text in SED
X line number
Range of row numbers x and y
/Pattern/query rows in the include Mode
/Pattern/query rows that contain two modes
/Pattern/, X queries matching mode rows on the specified row number
X,/pattern/query matched rows by row number and Pattern
X, Y! Query rows that do not contain rows X and Y
Sed edit command
P print text
Before matching metacharacters $, you must use the backslash \
= Print the row number. Use the-e option.
If both the row number and the matching row are printed, you must use two sed commands and use the-e option.
Additional Text
The A \ symbol can be used to specify one or more lines of text to be appended to the specified line. If no text placement position is specified, sed is placed after each row by default.
Create sed script file
Create a script file. The first action is as follows:
#! /Bin/sed-F ---- note the SED command interpretation line. The script searches for sed in this line to run the command, which is located in/bin
Insert text: insert text before a specified row. It only accepts one address.
Delete text: d
Replace commands with replace mode to replace the specified mode
An important function implemented by SED is to remove control characters from the files downloaded from another system.
1. Use S/-* // g to delete the horizontal line -----
2. Use/^ $ S/D to delete empty rows
3. Use $ d to delete the last row.
4. Use 1d to delete the first line
5. Use awk {print $1} to print the first column