Python Crawler Learning notes Regular expression of _python

Source: Internet
Author: User

Use of regular expressions

To learn the Python crawler, first you need to understand the use of regular expressions, so let's look at how to use them.

. The use of this time point is equivalent to a placeholder, you can match any one character, what do you mean? Just look at an example.

 Import re 
 content = "HelloWorld" 
 B = Re.findall (' W. ', content) 
 print B '

Notice, we first import the RE, this time we guess what the output is? Because. Equivalent to a placeholder, so the output of this time of course is wo.

* With the use of the above. Different, * can match the previous character any number of times, see an example

Content = "Helloworldhelloworld" 
B = Re.findall (' w* ', content) 
print B

The output at this time is [', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', '] ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ' is visible a list of lengths and matches of strings consistent, encountering a The matching characters are printed out.

. * The use of. * is a combination of use, it can be as much as possible to match content, such as the following example

Content = "Helloworldhelloworldworld" 
B = Re.findall (' he.*ld ', content) 
print B

It will output [' Helloworldhelloworldworld '], why does it not just print a helloworld, why is it all printed? This is a greedy algorithm, which means I have to find the longest qualifying content.

.*? In contrast to the above, this symbol will find the shortest possible qualifying content and then put it in a list, as shown below

Content = ' Xxhelloworldxxxxhelloworldxx ' 
b = re.findall (' xx.*?xx ', content) 
print B

The output of the result is [' xxhelloworldxx ', ' xxhelloworldxx '], visible, there is a xx in front of good annoying, how to remove it? Very simple, with parentheses, where is the bracket added?

Content = ' Xxhelloworldxxxxhelloworldxx ' 
b = Re.findall (' xx (. *?) xx ', content) 
print B

What we are talking about here is the case where the content does not contain line breaks, and what happens if the result of the line break is changed?

Content = ' Xxhelloworld xx ' ' 
b = Re.findall (' xx (. *?) xx ', content) 
print B

This time the output is an empty list, then how to do ah? If we write web crawler, the Web page source code must be more than a line ah, if we can not read the line, it is so embarrassing, of course, there are solutions ~

Content = ' Xxhelloworld xx ' ' 
b = Re.findall (' xx (. *?) xx ', Content,re. S) 
Print B

That's it, and there's a very handy technique for extracting numbers, as shown here

Content = ' xx123456 xx ' ' 
b = Re.findall (' (d+) ', Content,re. S) 
Print B

Crawl the picture link in the source code of the Web page and download

This article is only the first step of the web crawler, so the explanation is also relatively shallow, so now we first use the regular expression to implement a manual web crawler, what is manual? Is that we copy the Web page source code down, save in a TXT file, and then use regular expressions to filter information, and then to download.

First I searched the Linux desktop and found the following page

Right-click to view the network source code, press Ctrl+f search img src find the middle part of the copy, and paste into a TXT file,

Then we can use the above knowledge to extract the information we want, the source code is as follows

Import re import requests 
 F = open (' Source.txt ', ' r ') 
 HTML = f.read () 
 f.close () pattern 
 = '  
 

First of all we save the network source code TXT file, read, close the file stream, and then is to use regular expressions to extract the picture link, and finally using the requests in the Get () method to download the picture, note that this requests is not in Python itself, We need to download the specified file, then put it into the Python lib directory, download here, enter the site, press CTRL+F search keyword requests can see the following page

, we can see that we downloaded the. WHL suffix of the file, manually changed it to a. zip suffix, and then decompression, you can get two directories, the directory named requests copy paste to the directory mentioned above can be used.

All right, we're done here. Run the results

 C:python27python.exe e:/pythoncode/20160820/spider.py
 downloading:http://n1.itc.cn/img8/wb/smccloud/fetch/ 2015/07/04/112732422680200576.jpg
 downloading:http://n1.itc.cn/img8/wb/smccloud/fetch/2015/07/04/ 112640070563900918.JPG
 downloading:http://n1.itc.cn/img8/wb/smccloud/fetch/2015/07/04/112547718465744154. JPG
 downloading:http://n1.itc.cn/img8/wb/smccloud/fetch/2015/07/04/112455366330382227.jpg
 Downloading: Http://n1.itc.cn/img8/wb/smccloud/fetch/2015/07/04/112363014254719641.JPG
 downloading:http://n1.itc.cn/ Img8/wb/smccloud/fetch/2015/07/04/112270662197888742.jpg
 downloading:http://n1.itc.cn/img8/wb/smccloud/ Fetch/2015/07/04/112178310031994750.jpg
 downloading:http://n1.itc.cn/img8/wb/smccloud/fetch/2015/07/04/ 112085957910403853.JPG
 
 Process finished with exit code 0

This time the download was successful, go to our picture directory to see the downloaded pictures

The download was successful. Note that when you look for a Web page source code experiment, it is best not to have Chinese in the link, otherwise there may be garbled, because I learn Python is only a very short time, on the Chinese garbled problem, to deal with it is not so handy, so this will no longer explain, this article temporarily to paragraph, Have comments or questions can leave a message or private chat me.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.