Shell Scripts---grep, awk, sed tools

The grep command in the Grep:linux system is a powerful text-search tool that uses regular expressions to search for text and print matching lines. grep is a lookup filtering tool in which regular expressions find strings that conform to patterns in grep.

There are three variants of grep :

1.GRep: the standard grep command mainly discusses this format.

2.egrep: Extending grepto support basic and extended regular expressions

3.fgrep: Fast grep, allowing you to find strings instead of a pattern. The speed here is not as fast as the value

Format
grep [Options]

Main parameters
[Options] Main parameters:
-C: Outputs only the count of matching rows.
-I: Case insensitive (only for single-character).
-H: The file name is not displayed when querying multiple files.
-L: Only file names that contain matching characters are output when querying multiple files.
-N: Displays matching lines and line numbers.
-S: does not display error messages that do not exist or have no matching text.
-V: Displays all lines that do not contain matching text.

grep and regular expressions

Specifies that some special syntax represents a character class, a quantifier, and a positional relationship, and then uses these special grammars to represent a pattern together with ordinary characters, which is the regular expression (Regular expressions).

The extended specification for grep regular expressions is described above, and the basic specification has these syntaxes, just characters? +{}| () should be interpreted as ordinary characters, to indicate that the above special meaning needs to be added \ Escaped

Sed: streaming editor, which introduces the output of the previous program to the input of SED, and transforms it into another format output after a series of editing commands.

Sed is an online editor that processes a single line of content at a time. When processing, the currently processed rows are stored in a temporary buffer called pattern space, followed by the SED command to process the contents of the buffer, and after processing is done, the contents of the buffer are sent to the screen. Then the next line is processed, so it repeats until the end of the file. The file content does not change unless you use redirection to store the output. SED is mainly used to automatically edit one or more files, to simplify the repeated operation of the file, to write the conversion program and so on.

format : pattern/action where pattern is a regular expression, action is an edit operation. The SED program reads the pending file one line at a time, and if a line matches the pattern, the action is executed, and if a command has no pattern and only action, the action will be used for each row of the file to be processed.

Commands and Options
The sed command tells SED how to handle each input line specified by the address, and all input lines are processed if no address is specified.
Command
A\: Adds one or more rows after the current line. Multiple rows, except for the last row, need "\" to continue at the end of each line
C\: Replaces the text in the current line with the new text after this symbol. Multiple rows, except for the last row, need "\" to continue at the end of each line
I\: Inserts text before the current line. Multiple rows except the last row, the end of each line requires "\" To continue the row D Delete Row
H: Copy the contents of the pattern space to the staging buffer
H: Append the contents of the pattern space to the staging buffer
G: Copy the contents of the staging buffer to the schema space, overwriting the original content
G: Append the contents of the staging buffer to the schema space, appended to the original content
L: List non-printable characters
P: Print Line
N: Reads into the next input line and starts processing it from the next command instead of the first command
Q: End or exit sed
R: Read input line from file
! : Apply commands to all rows except the selected row
S: replace one string with another
G: Global substitution within a row
W: Write the selected line to the file
X: Swap the contents of the staging buffer with the pattern space
Y: Replace the character with another character (you cannot use the Y command on a regular expression)
Options
-E: Multiple edits, which are used when multiple SED commands are applied to the input line
-N: Cancel default output
-F: Specifies the file name of the SED script

1./pattern/p: Print lines matching pattern

2./PATTERN/D: Delete lines that match pattern

Note: The SED command does not modify the original file, and the Delete command only indicates that some lines do not print out the output, rather than deleting from the original file.

3./pattern/s/pattern1/pattern2/: Find the line that matches the pattern and replace the string with the first match pattern1 of the line with PATTERN2

4./pattern/s/pattern1/pattern2/g: Find the line that matches the pattern and replace all strings that match pattern1 with pattern2

5. Sed-i: The operation will modify the original file

6. Addressing

Sed-n ' 3p ' file #打印第三行

Sed ' 2,5d ' file # Delete the second line to line fifth

7. Exit status
The SED is not like grep, and its exit status is 0 regardless of whether the specified pattern is found. The exit status of SED is not 0 only if there is a syntax error in the command.

SED and regular expressions

Like grep, SED also supports special meta-characters for pattern lookups and substitutions. The difference is that the regular expression used by SED is the pattern enclosed by the slash line "/".

^: Beginning of Line locator:/^11/matches all lines starting with 11

$: Line Tail Locator:/11$/matches all lines ending with 11

&: Save the lookup string for reference in the replacement string: s/333333/*&*/g Symbol & Representative Find string.

X\{m\}: Continuous m x

X\{m,\}: At least m x

X\{m,n\}: At least m, but not more than n x

AWK: is a programming language tool for working with text. The language of the awk utility is similar to the Shell programming language in many ways, although awk has a syntax that belongs to itself entirely. When AWK was originally created, it was intended for text processing, and the language was based on the execution of a series of instructions as long as there was a pattern match in the input data. The utility scans each line in the file to find patterns that match what is given in the command line. If a match is found, proceed to the next programming step. If no match is found, continue with the next line. , awk is stronger than sed in that it can work with files not only in the unit of behavior but also as units. The default line delimiter for awk is newline, and the default column delimiter is contiguous spaces and tabs, but both the row and column separators can be customized.

The basic form of the awk command line is:
awk option ' script ' file1 file2 ...
awk option-f scriptfile file1 file2 ...
Like sed, awk handles files that can be redirected either by standard input or when command-line arguments are passed in, and the edit command can be passed directly to the command-line arguments, or you can specify a script file with the-f parameter, with the format of the edit command:
/pattern/{actions}
Condition{actions}

The utility divides each input row into records and fields. A record is a single line of input, and each record contains several fields. The default field delimiter is a space or tab, and the record delimiter is a newline. Although tabs and spaces are treated by default as field separators (multiple spaces remain as a delimiter), you can change the delimiter from a space to any other character. When AWK reads the input, the entire record is assigned to the variable. Each field is separated by a field delimiter, assigned to a variable, $ $, $ $, and so on. A line can essentially contain countless fields and access each field through the word Ge.

The automatic variable, $ $, is the first column, the second column, and so on, similar to the location parameter of the shell script, and $ A represents the entire current row.

In awk, two special expressions, begin and end, both can be used in the pattern, and the purpose of providing begin and end is to give the program an initial state and perform some finishing work after the program ends. Any action listed after begin (within {}) will be executed before awk begins scanning the input, and the actions listed after end will be executed after the input of the full part is scanned. Therefore, it is common to use begin to display variables and preset (initialize) variables, using end to output the final result.

How awk is invoked

1. Command-line mode
awk [-F field-separator] ' commands ' input-file (s)
Where commands is the true awk command, [-F domain delimiter] is optional. Input-file (s) is the file to be processed.

2.shell Script Mode
Insert all the awk commands into a file and make the awk program executable, and then awk command interpreter as the first line of the script, again by typing the script name to invoke.
Equivalent to the first line of the shell script: #!/bin/sh
Can be replaced by: #!/bin/awk-f

3. Insert all the awk commands into a separate file and then invoke:
Awk-f awk-script-file Input-file (s)

Where the-f option loads the awk script in Awk-script-file

Shell Scripts---grep, awk, sed tools