grep, sed, awk commands in the shell

grepCommand grepCommand Basic syntax

grepThe command is to print matching lines of text, all of which are called Global Search Regular Expression and print out of the line; The basic syntax is as follows:

grep[OPTIONS]PATTERN[FILE...]grep[OPTIONS][-e PATTERN | -f FILE][FILE...]

Wherein, OPTIONS represents the option, PATTERN indicates the matching pattern, the matching pattern can be a string, a variable, a regular expression, if the meaning space in the matching pattern, you need to use double quotation marks to match the pattern, to FILE represent a series of files; grep command to search the file for a line of text that satisfies the specified match pattern and print it out.

In the previous article "Regular expression", we can tell that the classification of regular expressions is:

• Basic Regular Expressions (basic Regular expression): Basic RegEx, short- BRE
• Extended Regular expression (Extended Regular expression): Extended RegEx, short ERE
• Perl Regular expression (perl Regular expression): Perl RegEx, abbreviated PCRE;

grepThe command supports different regular expressions based on different parameters:

• grepcommand does not follow any parameters (default), it means to use "BRE";
• grepCommand followed -E by parameters, it means to use "ERE";
• grepCommand followed -P by parameters, it means to use "PCRE";

Common OPTIONS options are shown in the following table:

Options	Description
-E	Specifies that pattern matching pattern is an extended regular expression
-F	Specifies that pattern match pattern is a fixed string
-G	Specifying pattern matching pattern as basic regular expression
-P	Specifies that pattern-matching mode is a Perl regular expression
-E PATTERN	Specify pattern as one or more search patterns
-F FILE	Specifies the FILE containing the search pattern
-I.	Match is ignore letter case
-V	Displays all lines of text that do not match the matching pattern
-W	Perform a word search
-X	Displays lines of text that exactly match the specified pattern without other characters
-Y	Same function as option-I
-C	Prints only the number of lines that match the lines of text and does not display matching content
-L	Displays only the file names of the matched lines of text and does not display matching content
-N	Lists all matching lines of text and displays line numbers
-R	Recursive search directory (current directory and subdirectories at all levels)
-S	Do not display error messages

Stream Editor sed sedCommand

sedcommand to read one row of data at a time from a file or standard input, copy the row data to a buffer, and then read the command line or script's edit subcommand to edit the lines of text in the buffer. Repeat this work until all lines of text have been processed.

sedThe command edits the file as a copy of the original file in the buffer, and does not affect the original file, but can save the result of the edits through output redirection.

sedThe basic syntax of the command is as follows:

sed [OPTION]... {script} [input-file]...

Common options OPTION are shown in the following table:

Options	Description
-N	Cancel default Output
-E Script	Allow multiple scripts to be executed
-F Script-file	To read a command from a script file
-I.	Modify the original file directly
-L N	Specify a line length of N
-R	Using an extended regular expression in a script
-S	Take the file as a separate file
-U	Minimal cache input and output

Text line positioning

sedThe command provides two ways to position text lines: line number positioning , regular expression positioning

Line number positioning

• Locates a specific line n : An n integer that represents the line to which the text is positioned n ;
• Locates a contiguous line n,m : n and is an m integer that represents the line of text that navigates to the starting behavior n and terminates m the behavior;
• Specify start line and step start~step : Start behavior start , step is step ;
• First 1 and last line $ : 1 represents the first row, $ indicating the last row;
• Specify a few lines following a row n,+x : The line from the n beginning of the line to the following row x ;

Regular expression positioning

sedThe syntax for using a regular expression to locate a text line is as follows:

#regexp 表示正则表达式/regexp/

sedCommon Operations for commands

sedCommon editing commands for commands are: print, delete, add, replace, and the basic syntax is as follows:

[address1[,address2]command[argument]

addressIt is called the positional parameter, which is the above row positioning; command is the sed provided subcommand, which is used to implement the editing operation; argument indicates the option parameter of the subcommand;

Print text

sedcommand to print the desired line of text through positional parameters and p commands, the syntax is as follows:

[address1[,address2]p

For example:

#! /bin/bash#输出第1~4行文本，注意1,3p之间是单引号‘1,3p‘ test.txt#输出以Lin开头的文本行‘/^Lin/ p‘ test.txt`echo"$result"

Replace text

sedThe command can replace the line of text with positional arguments and s subcommands, with the following syntax:

[address1[,address2]] s/pattern/replacemen/[flag]

flagTo replace the flags, different values affect s the behavior of the subcommands, and the flag commonly used values are shown in the following table:

Take value	Description
G	Global match, replaces all strings that conform to the rule in the text line
P	Replaces the first rule-compliant string and outputs the buffer to the standard output
W	Replace the first rule-compliant string and output the affected rows to a disk file
Decimal number N	Replaces the nth rule-compliant string in a text line
Empty	If you do not specify a flag value, replace the first rule-compliant string in the line of text

Delete text

sedIn the command, the line of text can be deleted by positional parameters and d subcommands, with the following syntax:

[address1[,address2]d

Add text

Append text
sedThe command can a append a line of text with positional arguments and subcommands, with the following syntax:

#将string插入到address1位置后面string

Insert Text
sedThe command can be i inserted into a line of text with positional arguments and subcommands, with the following syntax:

#将string插入到address1位置前面string

Combo command

sedThe command supports the combination of multiple subcommands, as follows:

• Use the -e options: -e option to combine multiple subcommands together, for example:

#! /bin/bash#将全部小写字母 e 替换成大写字母 E，并打印第 1 行到第 3 行文本行-e‘s/e/E/g‘-e‘1,3 p‘ test.txt`echo"$result"

• To ; execute multiple subcommands using semicolons:
  Its syntax format:

-e‘command1; command2...‘ filename

For example:

#! /bin/bash#将全部小写字母 e 替换成大写字母 E，并打印第 1 行到第 3 行文本行-e‘s/e/E/g; 1,3 p‘ test.txt`echo"$result"

• Use more than one subcommand for the same address:
  Its syntax format:

address{    command1    command2    command3    ...}#或[address] { command1;command2;command3;...}

sedScript file

Multiple subcommands can be written as a suffix named .sed sed script file, the sed script file content only need to list the individual subcommands, do not have to use quotation marks, if you write multiple subcommands on the same line, the different subcommands need to be separated by semicolons. sedthe format of the calling script file is as follows:

-f script

awkCommand awkBasic syntax for commands

awkThe command is a text processing tool whose basic syntax is as follows:

{ actions }

Where the pattern matching pattern represents the actions action to be performed, that is, the action is performed on the line of text that matches the pattern matching pattern, actions pattern and if the matching pattern is omitted, the operation is performed on all lines of text actions ; pattern
The matching pattern prints the output of a successful line of text, but it cannot be omitted at the same time pattern actions .

patternThe matching pattern can be one of the following:

• Regular expressions;
• Relationship expression;
• Mode 1 2 : Specifies the range of a row;
• BEGIN: Specifies the action that occurs before the first line of text is processed;
• END: Specifies the operation that is sent after the last line of text is read;

actionsCommands consist of one or more commands, functions, or expressions that are separated by a newline or semicolon and are in curly braces, typically in the following four cases:

• Variable or array assignment;
• Output command, for example printf or print ;
• Built-in functions;
• Flow control statements, such as if , while or for statements;

awkHow commands are executed:

• Execute from the command line: its syntax:

‘awk程序语句‘ 数据文件

• Execute awk script: its syntax:

#注意：awk脚本文件是后缀名为.awk的文件-f awk脚本文件 数据文件

• Executable script file: That is, specify the parser in the shell file as#! /bin/awk -f

awkThe matching mode of the command

The awk matching modes in the command include: relational expressions, regular expressions, blending modes, interval patterns, BEGIN patterns, END patterns, etc.

Relational expressions

awkThe relational operators in the command are shown in the following table:

operator	Description
>	Greater than
>=	Greater than or equal
<	Less than
<=	Less than or equal
==	Equals
!=	Not equal to
~	Matching operations: For example $1~/^A , a record that matches the first field begins with a character A
!~	Mismatch operation

Regular expressions

awkThe basic syntax for matching patterns with regular expressions is supported as follows:

#regexp 表示正则表达式/regexp/

Example: output a line of text beginning with the character U

#! /bin/bash#filename:test.sh‘/^u/ {print}‘ shell.md`echo"$result"#输出结果：unset variable_nameunset varuntiluntilunset-f 函数名

Mixed mode

In the awk match pattern of a command, you can use multiple expressions to compose a command through a logical operator, as shown in the following table:

operator	Description
&&	Logic and
||	Logical OR
!	Logical Non-

Interval mode

awkThe command supports interval mode with the following syntax:

pattern1, pattern2

For example: the first matching pattern is a line of text that begins with a string, and the one second pattern is a line 3 of text equal to the first field 14 , and all lines of text that match the two matching patterns will be output;

#!/bin/bash   #filename: test.sh  result= ' awk  '/^one/, $3==14 {print} '  Test.txt ' echo   " $result  "  #执行结果:  $ sh test.sh one 10  20  30 , 15  25  20  three 20  15  31  four 16  26  35  five 11  14  40   

BEGINMode

BEGINA pattern is a awk special built-in pattern of commands that awk perform operations before the data is read;

For example:

#!/usr/bin/awk-f#filename: test.shBEGIN {Print"beging operator."}/^ One/ , $3== -{print}#执行结果:$./test.sh test.txtbeging operator. One     Ten   -   - Both      the   -   -three    -   the   to Four     -   -   *Five     One   -   +

ENDMode

ENDA pattern is an action that is performed after the data has been processed, such as:

#! /usr/bin/awk -f# /usr/bin/env awk -f#filename:test.shBEGIN {    print "Beging operator."    print "===================="}/^one/ , $3==14 {print}END {    print "===================="    print "Ending operator."}#执行结果：$ ./test.sh test.txtBeging operator.====================one     10  20  30two     15  25  20three   20  15  31four    16  26  35five    11  14  40====================Ending operator.

awkVariable

awkUser-defined variables are generally defined in the BEGIN schema;

System built-in variables

awkThe common system built-in variables are shown in the following table:

variables	Description
$	Current record (the contents of the entire row are stored)
$1~ $n	The nth field of the current record, separated by FS between fields
Fs	Input field delimiter default is a space or tab
Nf	The number of fields in the current record, that is, how many columns
Nr	The number of records that have been read, is the line number, starting from 1, if there are multiple file words, this value is constantly accumulating
FNR	The current number of records, unlike NR, this value will be each file's own line number
Rs	Entered record delimiter, default to line break
OFS	Output field delimiter, default is also a space
ORS	Record delimiter for output, default to line break
FILENAME	The name of the current input file

Operator

awkCommands support commonly used operators: arithmetic operators, assignment operators, conditional operators, logical operations, relational operators, and so on;

Arithmetic operators

awkThe arithmetic operators directly supported by the command are: Plus +, minus-, multiply *, divide/, modulo operation%, exponential arithmetic ^;

Assignment operators

awkCommands commonly used assignment operators are: =, + =,-=, *=,/=,%=, ^=;

Conditional operators

awkThe conditional operator syntax for the command is as follows:

表达式?值1:值2

Logical Operation method

awkThe logical operators supported by the command:&&, | |,! ；

Relational operators

awkThe command supports relational operators:>, >=, <, <=, = =,! =, ~,!~;

awkFunction

awkA number of system functions are provided, and the user can also customize functions;

String functions

The commonly used string functions are:

function	Description
Index (string1, string2)	Returns the position of the first occurrence of string2 in string1
Length (String)	Returns the length of a string
Match (STRING,REGEXP)	Returns a string that conforms to the regexp substring
Split (String,array,seperator)	Separates string strings into multiple fields according to the delimiter Seperator, and stores them in an array of arrays
Sub (regexp,replacement,string)	Replaces string strings with the first regexp substring to replacement
Gsub (regexp,replacement,string)	Replace all substrings of regexp in string strings with replacement
SUBSTR (String,start,[length])	Length is truncated from the start position of string strings (if length is specified, it is truncated to string end)

Arithmetic functions

Commonly used arithmetic functions are: int (x), sqrt (x), exp (x), log (x), sin (x), cos (x), rand (x), Srand (x);

grep, sed, awk commands in the shell