Writing a web crawler in Python (iv): Introduction and application of opener and handler

Before you start, explain the two methods in Urllib2: info and Geturl

Urlopen returns an Answer object response (or Httperror instance) has two very useful methods info () and Geturl ()

1.geturl ():

This returns the real URL obtained, which is useful because the Urlopen (or the opener object) may be redirected. The URL you get may be different from the request URL.

As an example of a super link in everyone,

Let's build a urllib2_test10.py to compare the original URL and redirect links:

From URLLIB2 import request, Urlopen, Urlerror, httperror  
      

old_url = ' Http://rrurl.cn/b1UZuP '
req = Request (old_ URL)  
response = Urlopen (req)    
print ' old URL: ' + old_url  
print ' real URL: ' + response.geturl ()

After running, you can see the URL that the real link points to:

2.info ():

This returns the object's Dictionary object, which describes the obtained page condition. Typically, the server sends a specific header headers. The present is httplib. Httpmessage instance.

The classic headers contains "Content-length", "Content-type", and other content.

Let's build a urllib2_test11.py to test the application of info:

From URLLIB2 import request, Urlopen, Urlerror, httperror  
      
old_url = ' http://www.baidu.com '
req = Request (Old_url )  
response = Urlopen (req)    
print ' Info (): '
print response.info ()

The results of the run are as follows, and you can see the information about the page: