[Python] web crawler (4): Introduction of opener and handler and instance applications

Before proceeding, let's first explain the two methods in urllib2: info and geturl.

The response object response (or httperror instance) returned by urlopen has two useful methods: Info () and geturl ()

1. geturl ():

This returns the obtained real URL, which is useful because urlopen (or used by the opener object) may be redirected. The obtained URL may be different from the requested URL.

Take a hyperlink from a person as an example,

We will create a urllib2_test10.py file to compare the original URL with the redirection link:

from urllib2 import Request, urlopen, URLError, HTTPErrorold_url = 'http://rrurl.cn/b1UZuP'req = Request(old_url)response = urlopen(req)  print 'Old url :' + old_urlprint 'Real url :' + response.geturl()

After running the command, you can see the URL that the real link points:

2. Info ():

The dictionary object of the returned object, which describes the page information obtained. It is usually the specific headers sent by the server. Currently, this is an httplib. httpmessage instance.

The classic headers include "Content-Length", "Content-Type", and other content.

We will create a urllib2_test11.py to test the info application:

from urllib2 import Request, urlopen, URLError, HTTPErrorold_url = 'http://www.baidu.com'req = Request(old_url)response = urlopen(req)  print 'Info():'print response.info()

The running result is as follows:

The following describes two important concepts in urllib2: openers and handlers.

1. openers:

When you get a URL, you use an opener (an instance of urllib2.openerdirector ).

Normally, we use the default opener: urlopen.

But you can create personalized openers.

2. Handles:

Openers uses the processor handlers, and all the "heavy" Work is handled by handlers.

Each handlers knows how to enable URLs through a specific protocol, or how to handle various aspects of URL opening.

For example, HTTP redirection or HTTP cookies.

If you want to use a specific processor to obtain URLs, you will want to create an openers, such as an opener that can process cookies or an opener that does not redirect.

To create an opener, You Can instantiate an openerdirector,

Call. add_handler (some_handler_instance ).

Similarly, you can use build_opener, which is a more convenient function used to create opener objects. He only needs to call the handler function.
Build_opener adds several processors by default, but provides a quick way to add or update the default processor.

Other Processors handlers you may want to process proxy, verification, and other commonly used but a little special.

Install_opener is used to create (global) default opener. This indicates that calling urlopen will use the opener you installed.

The opener object has an open method.

This method can be directly used to obtain URLs like the urlopen function: you do not need to call install_opener, except for convenience.

After completing the above two content, let's take a look at the Basic Authentication content. Here we will use the opener and handler mentioned above.

Basic Authentication

To demonstrate how to create and install a handler, we will use httpbasicauthhandler.

When basic verification is required, the server sends a header (401 error code) request for verification. This specifies scheme and a 'realm', which looks like this: www-Authenticate: Scheme realm = "Realm ".

For example
WWW-Authenticate: Basic realm = "cpanel users"

The client must use a new request and contain the correct name and password in the request header.

This is "basic verification". To simplify this process, we can create an httpbasicauthhandler instance and let opener use this handler.

Httpbasicauthhandler uses a password management object to process URLs and realms to map user names and passwords.

If you know what realm is, you can use httppasswordmgr.

Usually people don't care what realm is. In this way, you can use the convenient httppasswordmgrwithdefaultrealm.

This will specify a default user name and password for the URL.

This will be provided when you provide another combination for a specific realm.

We specify none for the realm parameter and provide it to add_password to indicate this situation.

The URL at the highest level is the first URL that requires verification. The deeper URLs you pass to. add_password () will be the same.

After talking about so much nonsense, let's use an example to demonstrate the content mentioned above.

We will create a urllib2_test12.py to test the info application:

#-*-Coding: UTF-8-*-import urllib2 # create a password manager password_mgr = urllib2.httppasswordmgrwithdefaultrealm () # Add the user name and password top_level_url = "http://example.com/foo/" # If you know realm, we can use it to replace ''non ''. # define (none, top_level_url, username, password) password_mgr.add_password (none, top_level_url, 'why', '000000') # create a new handlerhandler = handler (password_mgr) # create an "opener" (openerdirector instance) opener = urllib2.build _ opener (handler) a_url = 'HTTP: // www.baidu.com/' # Use opener to obtain a urlopener. open (a_url) # Install opener. # Now all calls to urllib2.urlopen will use our opener. urllib2.install _ opener (opener)

Note: In the preceding example, we only provide our hhtpbasicauthhandler to build_opener.

The default openers have normal handlers: proxyhandler, unknownhandler, httphandler, httpdefaulterrorhandler, httpredirecthandler, ftphandler, filehandler, and httperrorprocessor.

The top_level_url in the code can be a complete URL (including "http:" and the host name and the optional port number ).

For example, http://example.com /.

It can also be an "authority" (that is, the host name and the optional include port number ).

For example, "example.com" or "example.com: 8080 ".

The latter contains the port number.