Linux Regular Expression

Linux Regular Expression Favorites

 

What is a regular expression?
A regular expression is a formula used to match a type of strings in a certain pattern. Many people are afraid to use it because they seem odd and complex. Unfortunately, this article cannot change this. However, after a few exercises, I began to think that these complex expressions are actually quite simple to write, and once you understand them, you can compress several hours of hard and error-prone Text Processing in minutes (or even seconds. Regular expressions are used by various text editing software and class libraries (for example, tools of rogue wave. h ++), script tools (such as awk/grep/SED) are widely supported, and interactive ide like Microsoft Visual C ++ is also starting to support it.
In the following sections, we will use some examples to explain the usage of regular expressions. Most examples are written based on the text replacement command in VI and the grep file search command, however, they are typical examples. The concepts can be used in SED, awk, Perl, and other programming languages that support regular expressions. You can look at the regular expressions section in different tools. Some examples of using regular expressions in other tools are provided. There is also a simple description of the text replacement command (s) in VI for reference.

Regular Expression Basics
A regular expression is composed of some common characters and metacharacters. Common characters include uppercase and lowercase letters and numbers, while metacharacters have special meanings. We will explain them below.
In the simplest case, a regular expression looks like a common query string. For example, the regular expression "testing" does not contain any metacharacters. It can match strings such as "testing" and "123testing", but cannot match "testing ".

To make good use of regular expressions, correct understanding of metacharacters is the most important thing. The following table lists all metacharacters and a brief description of them.

Metacharacter description

---------------------------

---------------------------

. Match any single character. For example, the regular expression R. T matches these strings: Rat, rut, and r t, but does not match root.
$ Match the row Terminator. For example, the regular expression weasel $ can match the end of the string "he's a weasel", but cannot match the string "they are a bunch of Weasels .".
^ Match the beginning of a row. For example, the regular expression ^ when in can match the start of the string "when in the course of human events", but cannot match "what and when in ".
* Matches zero or multiple characters that match exactly before it. For example, the regular expression. * means that it can match any number of characters.
/This is the reference house, used to match the metacharacters listed here as common characters. For example, the regular expression/$ is used to match the dollar sign, not the end of the line. Similarly, the regular expression/. is used to match the dot character, rather than any character wildcard.
[]
C1-c2
[^ C1-c2] matches any character in the brackets. For example, the regular expression R [aou] T matches rat, rot, and rut but does not match ret. You can use the hyphen (-) in brackets to specify the character range. For example, the regular expression [0-9] can match any number character. You can also specify multiple intervals, for example, the regular expression [A-Za-Z] can match any uppercase or lowercase letter. Another important usage is "exclude". To match characters other than the specified range, that is, the so-called supplement set, use the ^ character between the brackets on the left and the first character, for example, the regular expression [^ 269a-z] matches any character except 2, 6, 9, and all uppercase letters.
/</> Start (/<) and end (/>) of the match word ). For example, the regular expression/<The can match the "the" in the string "for the wise", but cannot match the "the" in the string "otherwise ". Note: This metacharacter is not supported by all software.
/(/) Defines the expression between/(and/) as "group ), and save the characters matching the expression to a temporary region (a regular expression can save up to 9 characters). They can be referenced with symbols from/1 to/9.
| Perform logical "or" (OR) operations on the two matching conditions. For example, the regular expression (him | her) matches "It belongs to him" and "It belongs to her", but does not match "it belongs to them .". Note: This metacharacter is not supported by all software.
+ Match one or more characters that match exactly before it. For example, the regular expression 9 + matches 9, 99, and 999. Note: This metacharacter is not supported by all software.
? Match 0 or 1 character that is exactly before it. Note: This metacharacter is not supported by all software.
/{I /}
/{I, j/} matches a specified number of characters, which are defined by the expression before it. For example, the regular expression a [0-9]/{3/} can match the character "A" followed by a string of exactly three numeric characters, such as A123 and a348, but does not match a1234. The regular expression [0-9]/{4, 6/} matches any four, five, or six consecutive numeric characters. Note: This metacharacter is not supported by all software.

---------------------------

The simplest metacharacters are vertices that can match any single character (note that the newline character is not included ). Assume that a file named test.txt contains the following lines:

He is a rat
He is in a rut
The food is rotten
I like root beer
We can use the grep command to test our regular expression. The grep command uses a regular expression to try to match each row of the specified file and display at least one row matching the expression. Command

1. Grep R. T test.txt

Search for the regular expression R. T in each row in the test.txt file and print the matched rows. The regular expression R. T matches an r followed by any character followed by a T. Therefore, it will match the rat and rut in the file, instead of the rot in rotten, because the regular expression is case sensitive. To match both uppercase and lowercase letters, use the character range metacharacters (square brackets ). The regular expression [RR] can match both R and R. Therefore, to match an upper or lower case r followed by any character followed by a T, use this expression: [RR]. T.
To match the characters at the beginning of a line, use the Escape Character (^), which is also called an insert character. For example, if you want to find the line starting with "he" in text.txt, you may first use a simple expression "he", but this will match the line in the third, so use the regular expression ^ he, it only matches h at the beginning of the row.

Sometimes it is easier to specify "match all except ×××". When the Escape Character (^) appears in square brackets, it indicates "exclude". For example, to match with HE, however, if we exclude T or S (that is, the and she), we can use [^ st] He.

You can use square brackets to specify multiple character ranges. For example, the regular expression [A-Za-Z] matches any letter, including uppercase and lowercase letters; the regular expression [A-Za-Z] [A-Za-Z] * matches a letter followed by 0 or multiple letters (uppercase or lowercase ). Of course, we can also use metacharacters + to do the same thing, that is, [A-Za-Z] +, it is equivalent to [A-Za-Z] [A-Za-Z. However, note that metacharacters + are not supported by all programs that support regular expressions. For more information, see the regular expression syntax.

To specify a specific number of matches, use braces (note that you must use a backslash to escape ). To match all instances of 100 and 1000 and exclude 10 and 10000, use 10/{2, 3/}. This regular expression matches the pattern followed by 2 or 3 0 after the number 1. A useful change in the usage of this metacharacter is to ignore the second number. For example, the regular expression 0/{3,/} matches at least three consecutive zeros.

Simple Example
Here are some representative and simple examples.

VI command function

---------------------------

---------------------------

: % S/* // G replace one or more spaces with one.
: % S/* $ // remove all spaces at the end of the line.
: % S/^ // Add a space on the header of each line.
: % S/^ [0-9] [0-9] * // remove all numeric characters from the beginning of the line.
: % S/B [aeio] g/bug/g change all bag, beg, big, and bog to bug.
: % S/t/([aou]/) g/h/1 t/g change all tags, Tog, and tug to hat, hot, and hug respectively (note the use of group and use/1 to reference the matched characters ).

Intermediate example (MAGIC manipulation)
Example 1
Change the instance of all methods Foo (A, B, C) to Foo (B, A, C ). Here, A, B, and C can be any parameter provided to the method Foo. That is to say, we need to achieve this conversion:

Before and after

Foo (10, 7, 2) Foo (7, 10, 2)
Foo (x + 13, y-2, 10) Foo (y-2, x + 13, 10)
Foo (Bar (8), X + Y + Z, 5) Foo (x + y + Z, bar (8), 5)

The following replacement command can achieve this magic:

1. : % S/Foo (/([^,] */),/([^,] */),/([^)] */) /Foo (/2,/1,/3)/g

Now let's scatter it for analysis. The basic idea of writing this expression is to find out the location of the three parameters in Foo () and its brackets. The first parameter is identified using this expression:/([^,] */). We can analyze it from the inside out:

[^,] Any character except comma
[^,] * 0 or multiple non-comma characters
/([^,] */) Mark these non-comma characters as/1, so that they can be referenced in the subsequent replacement mode expression
/([^,] */), We must find 0 or multiple non-comma characters followed by a comma, and the part of non-comma characters should be marked for future use.

It is now the best time to point out common errors using regular expressions. Why do we need to use an expression like [^,] * instead of simply writing it directly, such as. *, to match the first parameter? Suppose we use the pattern. * to match the string "10, 7, 2". Should it match "10," or "10, 7 ,"? To solve this ambiguity, regular expressions always follow the longest string. In the preceding example, the regular expressions are ,", obviously, we found two parameters instead of one we expected. Therefore, we need to use [^,] * to forcibly extract the part before the first comma.

We have analyzed this expression: Foo (/([^,] */), this section can be simply translated as "when you find Foo (mark the part after it until the first comma is/1 ". Then we use the same method to mark the second parameter as/2. The method for marking the third parameter is the same, but we need to search all the characters until the right parenthesis. We do not need to search for the third parameter, because we do not need to adjust its location, but this mode ensures that we only replace the Foo () method calls with three parameters, when Foo () is an overoading method, this explicit pattern is often relatively safe. Then, in the replacement part, we find the corresponding instance of Foo (), and then use the marked part to replace it. Yes, the first and second parameter exchange locations.

Example 2
Assume there is a CSV (comma separated value) file with some information we need, but the format is incorrect. The column sequence of the data is name, company name, and abbreviated state name, zip code. Now we want to reorganize the data so that it can be used in one of our software. The format is name, abbreviated state name-zip code, and company name. That is to say, we need to adjust the column order and combine two columns to form a new column. In addition, our software cannot accept any spaces (including spaces and tabs) before and after the comma. Therefore, we must remove all spaces before and after the comma.
Here are a few lines of our current data:

Bill Jones, HI-TEK Corporation, CA, 95011
Sharon Lee Smith, design works inreceivated, CA, 95012
B. amoperating, Hill Street Cafe, CA, 95013
Alexander weatherworth, the crafts store, CA, 95014

...
We want to make it look like this:

Bill Jones, CA 95011, HI-TEK Corporation
Sharon Lee Smith, CA 95012, design works inreceivated
B. Amos, CA 95013, Hill Street Cafe
Alexander weatherworth, CA 95014, the crafts store

...
We will use two regular expressions to solve this problem. The first column is moved and colated, and the second column is used to remove spaces.
The following is the first replacement command:

1. : % S // ([^,] */),/([^,] */),/([^,] */),/(. */)/1,/3/4,/2/

The method here is basically the same as Example 1. The first column (name) uses this expression to match:/([^,] */), that is, all the characters before the first comma, the name is marked with/1. The abbreviated company name and state name fields are marked as/2 and/3 in the same way, and the last field is marked /(. */) to match ("match all characters until the end of the line "). The Replacement Part references the content marked above for construction.
The following replacement command is used to remove spaces:

1. : % S/[/T] *, [/T] */,/g

Let's look at the decomposition: [/T] matches spaces/tabs, [/T] * matches 0 or multiple spaces/tabs, [/T] *, match 0 or multiple spaces/tabs with a comma. Finally, [/T] *, [/T] * matches 0 or more spaces/tabs, followed by a comma, followed by 0 or more spaces/tabs. In the replacement section, we simply replace everything we find with a comma. Here we use the optional G parameter at the end, which means to replace all matching strings in each line (instead of replacing the first matching string by default ).
Example 3
Assume that a multi-character segment is repeated, for example:

Billy tried really hard
Sally tried really hard
Tiequaltried really hard
Johnny tried really hard

You want to replace "really", "really", and any number of "really" strings that appear consecutively with a simple "very" (simple is good !), The following command:

1. : % S // (really/)/(really/) */very/

The above text will be changed:

Billy tried very hard
Sally tried very hard
Tiequaltried very hard
Johnny tried very hard

Expression/(really/) * matches 0 or multiple consecutive "really" (note that there is a space at the end), And/(really /) * matches one or more consecutive "really" instances.
Difficult examples (incredible hieroglyphics)
Coming soon.

---------------------------

Regular Expressions in different tools
OK. You are ready to use re (regular expressions, regular expression), but you are ready to use VI. Therefore, here we provide some examples of using RE in other tools. In addition, I will summarize the differences you may find when using RESS between different programs.
Of course, you can also use re in the Visual C ++ editor. Select edit-> replace, and then select the "Regular Expression" selection box. Find what input box corresponds to the pat1 section in the preceding VI command: % S/pat1/pat2/g, the replace input box corresponds to the pat2 part. However, to get the VI Execution scope and G options, you should use replace all or manually find next and replace, although VC can select a range of text, and then perform replacement in it, but in short is not as flexible and elegant as VI ).

Sed
Sed is short for stream editor. It is a commonly used file-and pipeline-based editing tool in Unix. You can obtain detailed information about sed in the manual.

This is an interesting sed script. We are processing a file called price.txt. Note that these edits do not change the source file. Sed only processes each row of the source file and displays the result in the standard output (of course, it is easy to use redirection for customization ):

Sed script description

---------------------------

---------------------------

1. Sed's/^ $/d' price.txt delete all empty rows
2. Sed's/^ [/T] * $/d' price.txt delete all rows that only contain spaces or tabs
3. Sed's/"// G' price.txt Delete All quotation marks

Awk
Awk is a programming language that can be used for complex analysis and processing of text data. You can obtain detailed information about awk in the manual. This odd name is the abbreviation of the author's surname (AHO, Weinberger, and kernighan ).
There are many good awk examples in the awk programming language in Aho, Weinberger, and kernighan books. Please do not let the following trivial script examples limit your understanding of awk's powerful capabilities. We use the same pattern to process the price.txt file. Like SED, awk only displays the result on the terminal.

Awk script description

---------------------------

---------------------------

Awk '$0 !~ /^ $/'Price.txt delete all empty rows
Awk 'nf> 0' price.txt awk a better way to delete all rows
Awk '$2 ~ /^ [JT]/{print $3} 'price.txt print all the third fields in the 'J' or 'T' header row
Awk '$2 !~ /[Mm] ISC/{print $3 + $4} 'price.txt for all rows whose second field does not contain 'misc' or 'misc, print the sum of columns 3rd and 4th (assumed to be a number)
Awk '$3 !~ /^ [0-9] + /. [0-9] * $/{print $0} 'price.txt print all rows where the third field is not a number. Here the number refers to D. D or D, where D is any number ranging from 0 to 9
Awk '$2 ~ /John | Fred/{print $0} 'price.txt. If the second field contains 'john' or 'fred ', print the entire line.

Grep
Grep is a program used to search by Re in one or more files or input streams. Its name programming language can be used to process files and pipelines. You can obtain complete grep information in the manual. This strange name is derived from a command of VI, G/RE/P, which means global regular expression print.
In the example below, we assume that the phone.txt file contains the following text, in the format of a surname with a comma, a name, a tab, and a phone number:

Francis, John 5-3871
Wong, Fred 4-4123
Jones, Thomas 1-4122.
Salazar, Richard 5-2522.

Grep command description

---------------------------

---------------------------

Grep '/T5-... 1' phone.txt prints all the lines whose phone numbers start with 5 and end with 1. Note that the tabs are represented by/t.
Grep '^ s [^] * r'phone.txt print all rows whose names start with S and whose names start with R
Grep '^ [JW]' phone.txt prints all rows whose names start with J or W.
Grep ',.../t' phone.txt prints all rows with a last name of 4 characters. Note that the tab is represented by/t.
Grep-V '^ [JW]' phone.txt prints all rows not starting with J or W
Grep '^ [M-Z]' phone.txt print all rows whose names start with any character between m and Z
Grep '^ [M-Z]. * [12]' phone.txt print all rows whose names start with any character between m and Z and whose dots end with 1 or 2

Egrep
Egrep is an extended version of grep. It supports more metacharacters in its regular expression. In the example below, we assume that the phone.txt file contains the following text, in the format of a surname with a comma, a name, a tab, and a phone number:

Francis, John 5-3871
Wong, Fred 4-4123
Jones, Thomas 1-4122.
Salazar, Richard 5-2522.
Egrep command description

---------------------------

---------------------------

Egrep '(John | Fred)' phone.txt prints all rows containing the name John or Fred
Egrep 'John | 22 $ | ^ W' phone.txt print all rows that contain John, end with 22, or end with W
Egrep 'net (work )? S 'report.txt find all lines that contain networks or nets from report.txt

---------------------------

Regular expression syntax support

1. Command or environment. [] ^ $ /(/)/{/}? + | ()
2. VI x
3. Visual c ++ x
4. Awk x
5. Sed x
6. TCL x
7. Ex x
8. Grep x
9. Egrep x
10. Fgrep x
11. Perl x

---------------------------

VI replacement command Introduction
VI replacement command:
: Ranges/pat1/pat2/g
Where
: This is the VI Command Execution interface.
Range is the execution range of the command. You can use percent sign (%) to indicate all rows, dot (.) To represent the current row, and dollar sign ($) to represent the last row. You can also use the row number. For example, 10, 20 indicates 10th to 20 rows ,., $ indicates the current row to the last row ,. + 2, $-5 indicates the last two rows of the current row until the bottom line of the full text, and so on.
S indicates a replacement command.

Pat1 is a regular expression to be searched. There are a lot of examples in this article.

Pat2: This is a regular expression used to convert a matching string into a pattern. There are a lot of examples in this article.
G indicates that the replacement will be performed on each matching string in the row. Otherwise, only the first matching string in the row will be replaced.