Python crawler (4)--Regular expression (i)

In the previous articles we used the Python urllib module and did some work on the Web page. Now introduce a very powerful tool-the regular expression. In telling the regular, I refer to the book "Proficient in Regular expression (third edition)---Jeffrey E.f.friedl", although the reference language of this book does not have Python, but there are a lot of similarities, you can read part of it.

Usually when we make the crawler to collect data, we need to filter the data transmitted by the server, for example, in the second article to download the link home page, we only need to address location and price. This is where the regular expression comes in handy. The so-called regular expression is a formal expression method that describes the structure pattern of a string. Initially, this set of methods was used to describe the regular text, and the later development of its function became very powerful. Python has joined the regular module re since the 1.5 release. On the one hand, the regular description is a sequence of objects, that is, the fundamental data structure of today's computer world, on the other hand, it has very strong structural description ability. Based on these two points, we need to learn about regular expressions.

This time we use a different module--requests to verify. First the PIP install requests, install the package. The use of requests seems to be simpler than Urllib:

1 #First we import this package2 Importrequsets3 4Url="http://www.17jita.com/tab/img/8088.html"5 6 #Get Method7Response=requests.get (URL)8 #Post Method9Response=requests.post (URL)Ten  One #other AResponse=request.put (URL) -Response=request.delete (URL) -Response=request.head (URL) theReponse=request.options (URL) -  - #Get content -Html=Response.text + Print(HTML)

About the Requests module, we can continue to learn in the process of later use. Let's look at the RE module based on the regular expression and then use re to match the content on the page.

Rules of Regular expressions

Regular expressions have the following metacharacters, each of which has a special meaning:.  ^  $  *  +  { }  [ ]  \   | ( ) 。 For example, a dot (.) indicates that any character other than a line break is matched, and the pipe character (|) is somewhat similar to a logical OR operation. We can check the regular manual on the http://tool.oschina.net/uploads/apidocs/jquery/regexp.html website. Help (re), we can see these methods: Match,fullmatch,search,split,findall ... such as

Let's give a few examples.  For example, we want to match the following data "[email protected] Advantage 314159265358 1892673 3.14 little Girl try_your_best [email protected] Python3 "

1, matching advantage, we use the search method to achieve, if the match succeeds it will return to the location.

As you can see, we only found the position of advantage first letter a span= (21,22). What if you want to match the whole word?

We can see the match to the advantage. Where (.) indicates a match for any character except for the body of the newline character, (+) that matches the preceding subexpression one or more times.

2. Match email address

The 0-9 inside the brackets indicates that 0123456789,a-z represents 26 letters, and {} indicates that the content is matched one or more times. This time we use FindAll (), he can find all the strings that satisfy the regular rules, and return the strings. Of course, the above rules can continue to be perfected so that they can match any mailbox.

In the next article we will resolve the problem of matching the IP address and crawl the available IP from the site.

Python crawler (4)--Regular expression (i)