Custom opener in Urllib2

The normal use of Python crawl web information, need to use URLLIB2, call Urllib2.urlopen (URL), you can get response feedback information, and then use Response.read () to get the source of the page.

The simplest pick-and-pack code:

Import urllib2  response = Urllib2.urlopen (' http://www.baidu.com/')  html = response.read ()  Print HTML

Here are two important concepts in URLLIB2: openers and handlers.

1.Openers:

When you get a URL you use a opener (a urllib2. Openerdirector instances).

Under normal circumstances, we use the default opener: through Urlopen.

But you can create a personality openers.

2.Handles:

Openers uses processor handlers, all "heavy" work is handled by handlers.

Each handlers knows how to open URLs through a specific protocol, or how to handle various aspects of the URL when it is opened.

When you use the agent to surf the Internet or other situations you need to create a opener, you can instantiate a openerdirector,

Then call. Add_handler (some_handler_instance).

Similarly, you can use Build_opener, which is a more convenient function for creating opener objects, and he only needs one function call at a time.
Build_opener adds several processors by default, but provides a quick way to add or update the default processor.

Other processor handlers you might want to handle proxies, validations, and other common but somewhat special situations.

Install_opener is used to create a (global) default opener. This indicates that calling Urlopen will use the opener you installed.

The opener object has an open method.

This method can be used to obtain URLs as directly as the Urlopen function: it is not usually necessary to call Install_opener, except for convenience.

Import urllib2  Proxy_handler = urllib2. Proxyhandler ({' http ': ' http://10.19.110.32:8080/'}) opener = Urllib2.build_opener (Proxy_handler) Urllib2.install_ Opener (opener) response = Urllib2.urlopen (' http://www.baidu.com/')  html = response.read ()  Print HTML

Import urllib2  Proxy_handler = urllib2. Proxyhandler ({' http ': ' http://10.19.110.32:8080/'}) opener = Urllib2.build_opener (proxy_handler) response = Opener.open (' http://www.baidu.com/')  html = response.read ()  Print HTML

　　

Custom opener in Urllib2