Python Crawler Primer (5)--Regular Expression Example tutorial

Source: Internet
Author: User
To be a crawler, the inevitable use of regular expressions, if it is a simple string processing, similar to split,substring and so on is enough, but involves a more complex match, of course, is the world, The following article mainly introduces the Python crawler of the regular expression of the relevant information, the need for friends can refer to.

Objective

Regular expression processing text like a blast sweeping the autumn leaves, most programming languages have built-in support for regular expressions, which are used in scenarios such as form validation, text extraction, substitution, and so on. The crawler system is inseparable from the regular expression, with good regular expression can often receive a multiplier effect.

Before we introduce regular expressions, let's take a look at a question, and the following text comes from a link to a Web page in the watercress, and I have reduced the content. Q: How do I extract all the email addresses in the text?

html = "" "  <style>   . qrcode-app{    Display:block;    Background:url (/pics/qrcode_app4@2x.png) no-repeat;   }  </style>  <p class= "Reply-doc content" >   <p class= "" >34613453@qq.com, thank you </p>   <p class= "" >30604259@qq.com trouble landlord </p>  </p>  <p class= "" >490010464@163.com<br/ > Thank you </p> "" "  

If you have not touched the regular expression, I think this will be helpless, not the regular, it seems that a better way to deal with, but we put aside this problem, after learning the regular expression before we consider how to solve.

The representation of a string

There are several representations of Python strings, strings that begin with u are called Unicode strings, which are not covered in this article, and you should also see these two types of notation:

>>> foo = "Hello" >>> bar = r "Hello"

The former is a regular string, and the latter r begins with the original string, what is the difference between the two? Because in the above example, they are all strings of ordinary text characters, there is no difference here, the following can prove


>>> foo is bartrue>>> foo = = Bartrue

But what happens if you include special characters in the string? Let's look at one more example:

>>> foo = "\ n" >>> bar = r "\ n" >>> foo, Len (foo) (' \ n ', 1) >>> Bar, Len (bar) (' \\n ', 2) > >> foo = = barfalse>>>

"\ n" is an escape character that represents a line break in ASCII. and r "\ n" is an original string, the original string is not escaped special characters, it is you see the literal meaning, "\" and "n" two characters of a string.

The definition of the original string can start with a lowercase R or a capital R, such as r "\b" or r "\b" are allowed. In Python, regular expressions are generally defined in the form of primitive strings, and why?

For example, for the character "\b", it has a special meaning in ASCII, which means the backspace key, and in the regular expression, it is a special meta-character, used to match the boundary of a word, in order to let the regular compiler to correctly express its meaning is necessary to use the original string, of course, you can use the backslash "\ "Escapes a regular defined string


>>> foo = "\\b" >>> bar = r "\b" >>> foo = = Bartrue

Regular Basic Introduction

Regular expressions consist of two characters, ordinary text characters and special characters (meta characters). Metacharacters has a special meaning in regular expressions, which makes the regular expression more expressive. For example, in the regular expression R "A.D", the characters ' a ' and ' d ' are ordinary characters, '. ' is a meta-character,. can refer to any character, it can match ' a1d ', ' a2d ', ' ACD ', its matching process is:

The Python built-in module re is a module dedicated to working with regular expressions.


>>> Rex = r "A.D" # Regular expression text >>> Original_str = "and" # Original text >>> pattern = Re.compile (Rex) # Regular expression pairs Like >>> m = Pattern.match (ORIGINAL_STR) # matches objects >>> m <_sre. Sre_match object at 0x101c85b28># is equivalent to >>> Re.match (r "A.D", "and") <_sre. Sre_match Object at 0x10a15dcc8>

If the original text string matches the regular expression, a match object is returned, and when it does not match, the match method returns none, and the form is validated by determining whether M is none.

Next, we need to learn more meta characters.

Basic Meta characters

    • .: matches any character except newline characters, for example: "A.C" can match "abc" exactly or match "abc" in "Abcef"

    • \: Escape character, so that special characters have the original meaning, for example: 1\.2 can match 1.2

    • [...] : Matches any one of the characters in the square brackets, for example: A[bcd]e can match Abe, Ace, Ade, and it also supports range operations, such as: A to Z can be expressed as "A-Z", 0 to 9 can be represented as "0-9", note that the special character in "[]" no longer has special meaning, is its literal meaning, For example: [. *] is a match. Or

    • [^ ...], the character set is reversed, indicating that the characters that appear in the brackets are not matched, for example: A[^bcd]e can match AEE, AFE, etc.


>>> Re.match (r "A.C", "ABC"). Group () ' ABC ' >>> Re.match (r "A.C", "ABCEF"). Group () ' ABC ' >>> Re.match (R "1\.2", "1.2"). Group () ' 1.2 ' >>> Re.match (r "A[0-9]b", "A2B"). Group () ' A2B ' >>> Re.match (r " A[0-9]b "," A5B11 "). Group () ' a5b ' >>> re.match (r" a[.*?) B "," A.B "). Group () ' A.B ' >>> re.match (r" abc[^\w] "," abc!123 "). Group () ' abc!

The Group method returns the part of the SUBSTRING (ABC) that matches the regular expression in the original string (ABCEF), which is to match the successful match method in advance to return the match object, and then the group method.

Preset meta characters

    • \w matches any single word character, including numbers and underscores, which is equivalent to [a-za-z0-9_], such as A\WC can match the ABC, ACC

    • \w matches any non-word character, as opposed to the \w operation, which is equivalent to [^a-za-z0-9_], for example: A\WC matches A!c

    • \s matches any white space character, space, carriage return, etc. are white space characters, for example: A\SC can be equipped with A\NC, here \ n means enter

    • \s matches any non-whitespace character

    • \d matches any number, which is equivalent to [0-9], for example: A\DC can match A1c, a2c ...

    • \d matches any non-numeric

Boundary matching

Boundary matching related symbols are specifically used to modify characters.

    • ^ matches the beginning of the character in front of the string, for example: ^abc represents a string that matches the beginning of a, followed by the BC, which can match the ABC

    • $ matches the end of the character at the end of the string, for example: hello$


>>> Re.match (r "^abc", "ABC"). Group () ' ABC ' >>> Re.match (r "^abc$", "ABC"). Group () ' ABC '

Repeat match

The previous metacharacters are all matched against a single character, and if you want the matching characters to repeat, such as matching the ID number, 18 bits in length, then you need to use a repeating match of metacharacters.

    • * Repeat matches 0 or more times

    • ? Repeat match 0 or one time

    • + Repeat match 1 or more times

    • {n} repeatedly matches n times

    • {N,} repeat matches at least n times

    • {n, m} repeatedly matches N to M times


# easy to match the ID number, the front 17 digits is the number, the last one can be a number or letter x>>> Re.match (r "\d{17}[\dx]", "42350119900101153X"). Group () ' 42350119900101153X ' # matches 5 to 12 of QQ number >>> Re.match (r "\d{5,12}$", "4235011990"). Group () ' 4235011990 '

Logical Branch

Match a fixed phone number, different regional rules are not the same, some local area code is 3-bit, the telephone is 8, some local area code is 4, the telephone is 7, the area code and the number used-separate, if the need to deal with this? At this point you need to use the logical branching conditional character |, which divides the expression into the left and right parts, first trying to match the part of the side, and if the match succeeds no longer matches the later part, which is the logical "or" relationship

# ABC|CDE can match either ABC or CDE, but preferentially match abc>>> Re.match (r "AA (ABC|CDE)", "AAABCCDE"). Group () ' Aaabc '

The 0\d{2}-\d{8}|0\d{3}-\d{7} expression starts with 0, matches either the 3-bit area code 8-digit number or the 4-bit area code 7-digit number

>>> Re.match (r "0\d{2}-\d{8}|0\d{3}-\d{7}", "0755-4348767"). Group () ' 0755-4348767 ' >>> Re.match (r " 0\D{2}-\D{8}|0\D{3}-\D{7} "," 010-34827637 "). Group () ' 010-34827637 '

Group

The matching rules described earlier are for a single character, and if you want to match multiple characters repeatedly, the answer is to use a subexpression (also called a grouping) to denote the grouping with parentheses "()", for example (ABC) {2} to match ABC two times, to match an IP address when it is available (\d{1,3 }\.) {3}\d{1,3}, because IP is made up of 4 arrays of 3 points, all, the preceding 3 groups of numbers and 3 points can be repeated 3 times as a group, and the last part is a string of 1 to 3 digits. such as: 192.168.0.1.

About grouping, the group method can be used to extract matching string groupings, by default it will match the entire expression as a No. 0 grouping, that is, the group () without parameters, or group (0), the first set of parentheses in the group is obtained by group (1), and so on

>>> m = Re.match (r "(\d+) (\w+)", "123ABC") # Group 0, matching the entire regular expression >>> m.group () ' 123abc ' #等价 >>> M.group (0) ' 123ABC ' # group 1, match first pair of brackets >>> M.group (1) ' 123 ' # Group 2, match second pair of brackets >>> M.group (2) ' ABC ' >>>

By grouping, we can extract the desired information from a string. In addition, groupings can also be obtained by specifying a name.

# The first group name is number# the second group name is char>>> m = Re.match (? p<number>\d+) (? p<char>\w+) "," 123abc ") >>> M.group (" number ") ' 123 ' # equivalent >>> m.group (1) ' 123 '

Greed and non-greed

By default, when a regular expression repeats a match, it is an insatiable pattern to match as many characters as possible when the entire expression is matched, which we call greedy mode. For example: R "A.*b" means that the match a begins with B, the middle can be a string of any number of characters, and if it matches the AAABCB, it matches the entire string.

>>> Re.match (r "A.*b", "AAABCB"). Group () ' AAABCB '

Sometimes we want to match as few as possible, what do we do? Just add a question mark after the quantifier "? "To match as few matches as possible, as in the example we just wanted to match Aaab, then we just need to modify the regular expression to r" A.*?b "

>>> Re.match (r "A.*?b", "AAABCB"). Group () ' Aaab ' >>>

Non-greedy patterns are used very frequently in crawler applications. For example, in the public "python Zen" once wrote a crawl site and convert it into a PDF file scene, on the Web page is related to the IMG tag element is a relative path, we need to replace it with an absolute path

>>> html = '  ' # Non-greedy mode matches two IMG tags # you can change to greedy mode to see how many >>> Rex = R '  ' >>> re.findall (Rex, HTML) ['/images/category.png ', '/images/js_framework.png ']>>>
>>> def Fun (match): ...  Img_tag = Match.group ()  ... src = match.group (1)  ... FULL_SRC = "Http://foofish.net" + src ...  New_img_tag = Img_tag.replace (src, full_src)  ... Return new_img_tag...>>> Re.sub (Rex, fun, html) 

The sub function can accept a function as the replacement target object, and the function return value is used to replace the part of the regular expression match, where I define the entire IMG tag as a regular expression R ', the group() return value is , and the return value of group (1) is/images/ Category.png, finally, I replace the relative path with an absolute path using the Replace method.

Here, you should have a preliminary understanding of the regular expression, and now I think you should be able to solve the issue of the beginning of the article.

The basic introduction of regular expressions is also here to the point, although the code example uses many of the methods in the RE module, but I have not formally introduced the module, considering the length of the article, I put this part in the next chapter, the next chapter will be the common methods of re introduction.

"Recommended"

1. Python Crawler Primer (4)--Detailed HTML text parsing library BeautifulSoup

2. Python Crawler Introduction (3)--using requests to build a knowledge API

3. Python crawler Primer (2)--http Library requests

4. Python crawler Primer (1)--Quick understanding of HTTP protocol

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.