Python crawlers: Use and examples of Python2.7 opener and handler

Before we start, let's explain two methods in Urllib2: info and Geturl

Response (or Httperror instance) of the reply object returned by Urlopen has two useful methods, info () and Geturl ()

1.geturl ():

This returns the actual URL obtained, which is useful because Urlopen (or opener objects used) may have redirects. The URL you get may be different from the request URL.

Take a hyperlink from everyone, for example,

Let's build a urllib2_test10.py to compare the original URL with the redirected link:

Import request, Urlopen, Urlerror, httperror      'http://rrurl.cn/b1UZuP' req = request (old_ URL) response =' oldurl:' +'Real URL:          

After running you can see the URL that the true link points to:

2.info ():

The Dictionary object that returns the object that describes the page condition that was obtained. Typically, the server sends a specific header headers. It is now httplib. Httpmessage instance.

The classic headers contains "Content-length", "Content-type", and other content.

Let's build a urllib2_test11.py to test the application of info:

Import Request, Urlopen, Urlerror, Httperror  2   'http://www.baidu.com4 req =5 Response ='Info ():         

Here are two important concepts in URLLIB2: openers and handlers.

1.Openers:

When you get a URL you use a opener (a urllib2. Openerdirector instances).

Under normal circumstances, we use the default opener: through Urlopen.

But you can create a personality openers.

2.Handles:

Openers uses processor handlers, all "heavy" work is handled by handlers.

Each handlers knows how to open URLs through a specific protocol, or how to handle various aspects of the URL when it is opened.

such as HTTP redirection or HTTP cookies.

If you want to get URLs with a specific processor, you'll want to create a openers, such as a opener that can handle cookies, or a opener that doesn't redirect.

To create a opener, you can instantiate a openerdirector,

Then call. Add_handler (some_handler_instance).

Similarly, you can use Build_opener, which is a more convenient function for creating opener objects, and he only needs one function call at a time.
Build_opener adds several processors by default, but provides a quick way to add or update the default processor.

Other processor handlers you might want to handle proxies, validations, and other common but somewhat special situations.

Install_opener is used to create a (global) default opener. This indicates that calling Urlopen will use the opener you installed.

The opener object has an open method.

This method can be used to obtain URLs as directly as the Urlopen function: it is not usually necessary to call Install_opener, except for convenience.

After saying the above two things, let's take a look at the basic certification content, here will use the above mentioned opener and handler.

Basic Authentication Fundamental Validation

In order to demonstrate the creation and installation of a handler, we will use Httpbasicauthhandler.

When basic authentication is required, the server sends a header (401 error code) to request validation. This specifies SCHEME and a ' realm ', which looks like this: Www-authenticate:scheme realm= "Realm".

For example
Www-authenticate:basic realm= "CPanel Users"

The client must use the new request and include the correct name and password in the request header.

This is "basic verification", in order to simplify this process, we can create an instance of Httpbasicauthhandler, and let opener use this handler.

Httpbasicauthhandler uses a password-managed object to process URLs and realms to map user names and passwords.

If you know what realm is (from the server), you can use Httppasswordmgr.

Often people don't care what realm is. In that case, you can use the convenient httppasswordmgrwithdefaultrealm.

This will specify a default user name and password for the URL.

This will be provided when you provide a different combination for a particular realm.

We indicate this by specifying none for the realm parameter to be provided to Add_password.

The highest-level URL is the first one that requires validation. The more profound URLs you pass on to. Add_password () will be equally appropriate.

Having said so much nonsense, here's an example to illustrate what is said above.

Let's build a urllib2_test12.py to test the application of info:

1#-*-Coding:utf-8-*-2ImportUrllib234#Create a password manager5 Password_mgr =Urllib2. Httppasswordmgrwithdefaultrealm ()67#Add user name and password89 Top_level_url ="http://example.com/foo/"1011#If we know realm, we can use him instead of ' None '.12#Password_mgr.add_password (None, top_level_url, username, password)Password_mgr.add_password (None, Top_level_url,‘Why‘,‘1223‘)1415#A new handler is created.Handler =Urllib2. Httpbasicauthhandler (Password_mgr)1718#Create "opener" (Openerdirector instance)Opener =Urllib2.build_opener (Handler) a_url = 'http://www.baidu.com/'    for a URL using opener  Opener.ope N (a_url)   # installs opener.   Now all calls to Urllib2.urlopen will be used by our opener.  Urllib2.install_opener (opener)            

Note: The above example we only provide our hhtpbasicauthhandler to Build_opener.

The default openers has a normal condition of handlers:proxyhandler,unknownhandler,httphandler,httpdefaulterrorhandler, HTTPRedirectHandler, Ftphandler, Filehandler, Httperrorprocessor.

The Top_level_url in the code can actually be a full URL (containing "http:" and a host name and an optional port number).

For example: http://example.com/.

It can also be a "authority" (that is, host name and optional include port number).

For example: "example.com" or "example.com:8080".

The latter contains the port number

Python crawlers: Use and examples of Python2.7 opener and handler