Python crawler _ getting started with python Crawler

Python crawler _ getting started with python Crawler

I thought there was nothing to write about, because this is a search engine on the internet, but crawlers are a big highlight of python after all. I'm sorry for this.
For the basic point, python2 requires two modules: urllib and urllib2, but there are actually re
First, we will introduce some common functions of the module.
Urllib. urlopen ('HTTP: // xxx. xxx. XXX') # open a website, which is similar to open.
Urllib2.Request (url) # resolve the url. This can be omitted. It is not very well understood. Some functions, such as adding head headers, also need to be used.
Urllib. urlretrieve (url, filename) # download the content provided by the url and save it with filename.
Here is an example of a hummingbird crawling image. The pseudocode is as follows:

1. url Parsing 2. Enable url3 and read the url, that is, read () 4. Use re. findall to find all the addresses related to the image. Here, only jpg5 and loop download are available.

As shown in the figure, the image link format is src = "http: // index_url/page_num/image_name.jpg? XXXXXXX ", if you need to download it, it must be a red circle, that is, http: // index_url/page_num/image_name.jpg
After the analysis, the subsequent things are easy to handle. The code below

Import urllibimport urllib2import re # process the address and obtain the image address def get_image_url (url): # url_format = urllib2.Request (url) #1 url_open = urllib. urlopen (url) #2 url_read = url_open.read () #3 re_value = re. compile ('(? <= Src \ = \").*? \. Jpg ') image_url_list = re. findall (re_value, url_read) #4 return image_url_list # This function is specially used for downloading. The first two rows delete all the content in/before the image connection, leave the file name behind to save the file. try not to mention it. If you are not clear, please go back to the fault tolerance def down_image (image_url): rev = '^. */'file_name = re. sub (rev, '', image_url) try: urllib. urlretrieve (image_url, file_name) failed T: print 'Download % s fail '% image_url else: print 'Download % s successed '% image_urlif _ name _ =' _ main _ ': url = 'HTTP: // bbs.fengniao.com/forum/10384633.html' image_url_list = get_image_url (url) for image_url in image_url_list: down_image (image_url) #5

Sleepy, go to bed ..... If you have time to talk about page flip or something, you can crawl the website.