Python implements a simple crawler

Now the online resources are particularly rich, especially if you see some of the site's good-looking pictures will want to save, but it is troublesome to need one click to download, but if we use the program to deal with, the problem will become very simple, only need to run the program to get all the pictures on the entire page.

Below we can look at how to use Python to achieve a simple Web page image to get the crawler applet.

First, how to get the whole page?

getpage.py

Import urllibdef getpage (URL):    page = urllib.urlopen (URL)    content = Page.read ()    return contentcontent = GetPage ("http://tieba.baidu.com/p/2510089409")

The first thing to do is import the Urllib module, which helps us to open any resource through the URL

Urlopen (URL, [, data])--To open a Web page according to the URL, according to the parameters to distinguish the post or get

Read (): Read all information on the entire page

Second, how to get only pictures?

savephotos.py

Import urllibimport redef getpage (URL):    page = urllib.urlopen (URL)    content = Page.read ()    return contentdef Getphotos (content):    reg = R ' src= "(. +?\.jpg)" Pic_ext '    Imgre = Re.compile (reg)    imglist = Re.findall (Imgre, Content)    x = 0    for Imgurl in imglist:        urllib.urlretrieve (Imgurl, './photos/%s.jpg '% x)        x+=1content = GetPage ("http://tieba.baidu.com/p/2510089409") print Getphotos (content)

The regular expression. Here we use regular expressions to filter the information we get to the Web page.

Import Urllib and re modules

Parsing a statement in a method getphotos.py

The first step: Create a regular expression filter rule;

Step two: Compile the regular expression

Step three: Save the rule-compliant file as a picture in imglist

Fourth step: Save the picture locally using the Urlretrieve () method. Where the second parameter setting saves the file location.

This makes it possible to implement a Python simple crawler that gets web images.

Climb to the picture, as follows:

　

Python implements a simple crawler