grep
Command
grep
Command Basic syntax
grep
The command is to print matching lines of text, all of which are called Global Search Regular Expression and print out of the line; The basic syntax is as follows:
grep[OPTIONS]PATTERN[FILE...]grep[OPTIONS][-e PATTERN | -f FILE][FILE...]
Wherein, OPTIONS
represents the option, PATTERN
indicates the matching pattern, the matching pattern can be a string, a variable, a regular expression, if the meaning space in the matching pattern, you need to use double quotation marks to match the pattern, to FILE
represent a series of files; grep
command to search the file for a line of text that satisfies the specified match pattern and print it out.
In the previous article "Regular expression", we can tell that the classification of regular expressions is:
- Basic Regular Expressions (basic Regular expression): Basic RegEx, short- BRE
- Extended Regular expression (Extended Regular expression): Extended RegEx, short ERE
- Perl Regular expression (perl Regular expression): Perl RegEx, abbreviated PCRE;
grep
The command supports different regular expressions based on different parameters:
grep
command does not follow any parameters (default), it means to use "BRE";
grep
Command followed -E
by parameters, it means to use "ERE";
grep
Command followed -P
by parameters, it means to use "PCRE";
Common OPTIONS
options are shown in the following table:
Options |
Description |
-E |
Specifies that pattern matching pattern is an extended regular expression |
-F |
Specifies that pattern match pattern is a fixed string |
-G |
Specifying pattern matching pattern as basic regular expression |
-P |
Specifies that pattern-matching mode is a Perl regular expression |
|
|
-E PATTERN |
Specify pattern as one or more search patterns |
-F FILE |
Specifies the FILE containing the search pattern |
-I. |
Match is ignore letter case |
-V |
Displays all lines of text that do not match the matching pattern |
-W |
Perform a word search |
-X |
Displays lines of text that exactly match the specified pattern without other characters |
-Y |
Same function as option-I |
|
|
-C |
Prints only the number of lines that match the lines of text and does not display matching content |
-L |
Displays only the file names of the matched lines of text and does not display matching content |
-N |
Lists all matching lines of text and displays line numbers |
-R |
Recursive search directory (current directory and subdirectories at all levels) |
-S |
Do not display error messages |
Stream Editor
sed
sed
Command
sed
command to read one row of data at a time from a file or standard input, copy the row data to a buffer, and then read the command line or script's edit subcommand to edit the lines of text in the buffer. Repeat this work until all lines of text have been processed.
sed
The command edits the file as a copy of the original file in the buffer, and does not affect the original file, but can save the result of the edits through output redirection.
sed
The basic syntax of the command is as follows:
sed [OPTION]... {script} [input-file]...
Common options OPTION
are shown in the following table:
Options |
Description |
-N |
Cancel default Output |
-E Script |
Allow multiple scripts to be executed |
-F Script-file |
To read a command from a script file |
-I. |
Modify the original file directly |
-L N |
Specify a line length of N |
-R |
Using an extended regular expression in a script |
-S |
Take the file as a separate file |
-U |
Minimal cache input and output |
Text line positioning
sed
The command provides two ways to position text lines: line number positioning , regular expression positioning
Line number positioning
- Locates a specific line
n
: An n
integer that represents the line to which the text is positioned n
;
- Locates a contiguous line
n,m
: n
and is an m
integer that represents the line of text that navigates to the starting behavior n
and terminates m
the behavior;
- Specify start line and step
start~step
: Start behavior start
, step is step
;
- First
1
and last line $
: 1
represents the first row, $
indicating the last row;
- Specify a few lines following a row
n,+x
: The line from the n
beginning of the line to the following row x
;
Regular expression positioning
sed
The syntax for using a regular expression to locate a text line is as follows:
#regexp 表示正则表达式/regexp/
sed
Common Operations for commands
sed
Common editing commands for commands are: print, delete, add, replace, and the basic syntax is as follows:
[address1[,address2]command[argument]
address
It is called the positional parameter, which is the above row positioning; command
is the sed
provided subcommand, which is used to implement the editing operation; argument
indicates the option parameter of the subcommand;
Print text
sed
command to print the desired line of text through positional parameters and p
commands, the syntax is as follows:
[address1[,address2]p
For example:
#! /bin/bash#输出第1~4行文本,注意1,3p之间是单引号‘1,3p‘ test.txt#输出以Lin开头的文本行‘/^Lin/ p‘ test.txt`echo"$result"
Replace text
sed
The command can replace the line of text with positional arguments and s
subcommands, with the following syntax:
[address1[,address2]] s/pattern/replacemen/[flag]
flag
To replace the flags, different values affect s
the behavior of the subcommands, and the flag
commonly used values are shown in the following table:
Take value |
Description |
G |
Global match, replaces all strings that conform to the rule in the text line |
P |
Replaces the first rule-compliant string and outputs the buffer to the standard output |
W |
Replace the first rule-compliant string and output the affected rows to a disk file |
Decimal number N |
Replaces the nth rule-compliant string in a text line |
Empty |
If you do not specify a flag value, replace the first rule-compliant string in the line of text |
Delete text
sed
In the command, the line of text can be deleted by positional parameters and d
subcommands, with the following syntax:
[address1[,address2]d
Add text
Append text
sed
The command can a
append a line of text with positional arguments and subcommands, with the following syntax:
#将string插入到address1位置后面string
Insert Text
sed
The command can be i
inserted into a line of text with positional arguments and subcommands, with the following syntax:
#将string插入到address1位置前面string
Combo command
sed
The command supports the combination of multiple subcommands, as follows:
- Use the
-e
options: -e
option to combine multiple subcommands together, for example:
#! /bin/bash#将全部小写字母 e 替换成大写字母 E,并打印第 1 行到第 3 行文本行-e‘s/e/E/g‘-e‘1,3 p‘ test.txt`echo"$result"
- To
;
execute multiple subcommands using semicolons:
Its syntax format:
-e‘command1; command2...‘ filename
For example:
#! /bin/bash#将全部小写字母 e 替换成大写字母 E,并打印第 1 行到第 3 行文本行-e‘s/e/E/g; 1,3 p‘ test.txt`echo"$result"
- Use more than one subcommand for the same address:
Its syntax format:
address{ command1 command2 command3 ...}#或[address] { command1;command2;command3;...}
sed
Script file
Multiple subcommands can be written as a suffix named .sed
sed
script file, the sed
script file content only need to list the individual subcommands, do not have to use quotation marks, if you write multiple subcommands on the same line, the different subcommands need to be separated by semicolons. sed
the format of the calling script file is as follows:
-f script
awk
Command
awk
Basic syntax for commands
awk
The command is a text processing tool whose basic syntax is as follows:
{ actions }
Where the pattern
matching pattern represents the actions
action to be performed, that is, the action is performed on the line of text that matches the pattern
matching pattern, actions
pattern
and if the matching pattern is omitted, the operation is performed on all lines of text actions
; pattern
The matching pattern prints the output of a successful line of text, but it cannot be omitted at the same time pattern
actions
.
pattern
The matching pattern can be one of the following:
- Regular expressions;
- Relationship expression;
- Mode
1
2
: Specifies the range of a row;
BEGIN
: Specifies the action that occurs before the first line of text is processed;
END
: Specifies the operation that is sent after the last line of text is read;
actions
Commands consist of one or more commands, functions, or expressions that are separated by a newline or semicolon and are in curly braces, typically in the following four cases:
- Variable or array assignment;
- Output command, for example
printf
or print
;
- Built-in functions;
- Flow control statements, such as
if
, while
or for
statements;
awk
How commands are executed:
- Execute from the command line: its syntax:
‘awk程序语句‘ 数据文件
- Execute
awk
script: its syntax:
#注意:awk脚本文件是后缀名为.awk的文件-f awk脚本文件 数据文件
- Executable script file: That is, specify the parser in the
shell
file as#! /bin/awk -f
awk
The matching mode of the command
The awk
matching modes in the command include: relational expressions, regular expressions, blending modes, interval patterns, BEGIN
patterns, END
patterns, etc.
Relational expressions
awk
The relational operators in the command are shown in the following table:
operator |
Description |
> |
Greater than |
>= |
Greater than or equal |
< |
Less than |
<= |
Less than or equal |
== |
Equals |
!= |
Not equal to |
~ |
Matching operations: For example $1~/^A , a record that matches the first field begins with a character A |
!~ |
Mismatch operation |
Regular expressions
awk
The basic syntax for matching patterns with regular expressions is supported as follows:
#regexp 表示正则表达式/regexp/
Example: output a line of text beginning with the character U
#! /bin/bash#filename:test.sh‘/^u/ {print}‘ shell.md`echo"$result"#输出结果:unset variable_nameunset varuntiluntilunset-f 函数名
Mixed mode
In the awk
match pattern of a command, you can use multiple expressions to compose a command through a logical operator, as shown in the following table:
operator |
Description |
&& |
Logic and |
|| |
Logical OR |
! |
Logical Non- |
Interval mode
awk
The command supports interval mode with the following syntax:
pattern1, pattern2
For example: the first matching pattern is a line of text that begins with a string, and the one
second pattern is a line 3
of text equal to the first field 14
, and all lines of text that match the two matching patterns will be output;
#!/bin/bash #filename: test.sh result= ' awk '/^one/, $3==14 {print} ' Test.txt ' echo " $result " #执行结果: $ sh test.sh one 10 20 30 , 15 25 20 three 20 15 31 four 16 26 35 five 11 14 40
BEGIN
Mode
BEGIN
A pattern is a awk
special built-in pattern of commands that awk
perform operations before the data is read;
For example:
#!/usr/bin/awk-f#filename: test.shBEGIN {Print"beging operator."}/^ One/ , $3== -{print}#执行结果:$./test.sh test.txtbeging operator. One Ten - - Both the - -three - the to Four - - *Five One - +
END
Mode
END
A pattern is an action that is performed after the data has been processed, such as:
#! /usr/bin/awk -f# /usr/bin/env awk -f#filename:test.shBEGIN { print "Beging operator." print "===================="}/^one/ , $3==14 {print}END { print "====================" print "Ending operator."}#执行结果:$ ./test.sh test.txtBeging operator.====================one 10 20 30two 15 25 20three 20 15 31four 16 26 35five 11 14 40====================Ending operator.
awk
Variable
awk
User-defined variables are generally defined in the BEGIN
schema;
System built-in variables
awk
The common system built-in variables are shown in the following table:
variables |
Description |
$ |
Current record (the contents of the entire row are stored) |
$1~ $n |
The nth field of the current record, separated by FS between fields |
Fs |
Input field delimiter default is a space or tab |
Nf |
The number of fields in the current record, that is, how many columns |
Nr |
The number of records that have been read, is the line number, starting from 1, if there are multiple file words, this value is constantly accumulating |
FNR |
The current number of records, unlike NR, this value will be each file's own line number |
Rs |
Entered record delimiter, default to line break |
OFS |
Output field delimiter, default is also a space |
ORS |
Record delimiter for output, default to line break |
FILENAME |
The name of the current input file |
Operator
awk
Commands support commonly used operators: arithmetic operators, assignment operators, conditional operators, logical operations, relational operators, and so on;
Arithmetic operators
awk
The arithmetic operators directly supported by the command are: Plus +, minus-, multiply *, divide/, modulo operation%, exponential arithmetic ^;
Assignment operators
awk
Commands commonly used assignment operators are: =, + =,-=, *=,/=,%=, ^=;
Conditional operators
awk
The conditional operator syntax for the command is as follows:
表达式?值1:值2
Logical Operation method
awk
The logical operators supported by the command:&&, | |,! ;
Relational operators
awk
The command supports relational operators:>, >=, <, <=, = =,! =, ~,!~;
awk
Function
awk
A number of system functions are provided, and the user can also customize functions;
String functions
The commonly used string functions are:
function |
Description |
Index (string1, string2) |
Returns the position of the first occurrence of string2 in string1 |
Length (String) |
Returns the length of a string |
Match (STRING,REGEXP) |
Returns a string that conforms to the regexp substring |
Split (String,array,seperator) |
Separates string strings into multiple fields according to the delimiter Seperator, and stores them in an array of arrays |
Sub (regexp,replacement,string) |
Replaces string strings with the first regexp substring to replacement |
Gsub (regexp,replacement,string) |
Replace all substrings of regexp in string strings with replacement |
SUBSTR (String,start,[length]) |
Length is truncated from the start position of string strings (if length is specified, it is truncated to string end) |
Arithmetic functions
Commonly used arithmetic functions are: int (x), sqrt (x), exp (x), log (x), sin (x), cos (x), rand (x), Srand (x);
grep, sed, awk commands in the shell