[Python] web crawler (4): Opener, Handler, and openerhandler

[Python] web crawler (4): Opener, Handler, and openerhandler

Before proceeding, let's first explain the two methods in urllib2: info and geturl.

The response object response (or HTTPError instance) returned by urlopen has two useful methods: info () and geturl ()

1. geturl ():

Geturl () returns the obtained real URL, which is useful because urlopen (or the opener object) may be redirected. The obtained URL may be different from the requested URL.

Take a hyperlink from a person as an example,

We will create a urllib2_test10.py file to compare the original URL with the redirection link:

[Python]View plaincopy

1. From urllib2 import Request, urlopen, URLError, HTTPError
4. Old_url = 'HTTP: // rrurl.cn/b1UZuP'
5. Req = Request (old_url)
6. Response = urlopen (req)
7. Print 'old url: '+ old_url
8. Print 'real url: '+ response. geturl ()

After running the command, you can see the URL that the real link points:

2. info ():

Info () returns the dictionary object of the object, which describes the page information obtained. It is usually the specific headers sent by the server.

The classic headers include "Content-length", "Content-type", and other Content.

We will create a urllib2_test11.py to test the info application:

[Python]View plaincopy

1. From urllib2 import Request, urlopen, URLError, HTTPError
3. Old_url = 'HTTP: // www.baidu.com'
4. Req = Request (old_url)
5. Response = urlopen (req)
6. Print 'info ():'
7. Print response.info ()

The running result is as follows:

The following describes two important concepts in urllib2: Openers and Handlers.

I. opener

The urllib2.urlopen () function does not support authentication, cookie, or other advanced HTTP functions. To support these functions, you must use build_opener () (which can be used to allow python programs to simulate browser access, so you can understand it ~) Function to create a custom Opener object.

Usage:

build_opener([handler1 [ handler2, ... ]])

Handler is a Handler instance. HTTPBasicAuthHandler, HTTPCookieProcessor, and ProxyHandler are commonly used.
The objects returned by build_opener () have the open () method, which is the same as the functions of the urlopen () function.
To modify the http header, you can use:

import urllib2opener = urllib2.build_opener()opener.addheaders = [('User-agent', 'Mozilla/5.0')]opener.open('http://www.example.com/')

Install_opener (opener) installs different opener objects as the global opener used by urlopen.
The HTTPBasicAuthHandler () handler can use add_password () to set the password.

h.add_password(realm,uri,user,passwd)

Realm is the name or description associated with the verification, depending on the remote server. Uri is the base URL. User and passwd specify the user name and password respectively.

import urllib2auth=urllib2.HTTPBasicAuthHandler()auth.add_password('Administrator','http://www.example.com','Dave','123456')opener=urllib2.build_opener(auth)u=opener.open('http://www.example.com/evilplan.html')

Cookie processing (HTTPCookieProcessor)

import urllib2,cookielibcookie=cookielib.CookieJar()cookiehand=urllib2.HTTPCookieProcessor(cookie)opener=urllib2.build_opener(cookiehand)

Proxy (ProxyHandler)

The ProxyHandler (proxies) parameter proxies is a dictionary that maps protocol names (http, ftp) and so on to the URL of the corresponding proxy server.

proxy=ProxyHandler({'http':'http://someproxy.com:8080'})auth=HTTPBasicAuthHandler()auth.add_password()opener=build_opener(auth,proxy)

You can also use a proxy in urlopen.

import urllib2  proxy = 'http://%s:%s@%s' % ('userName', 'password', 'proxy')  inforMation = urllib2.urlopen("http://www.example.com", proxies={'http':proxy})  

2. Handler

Handler is used to process URLs, such as HTTP redirection and HTTP cookies.
If you want to create a specific openers, such as obtaining an opener that can process cookies or obtaining an opener without redirection, you need to use a Custom handler.

Basic Authentication: In order to demonstrate how to create and install a handler, we will use HTTPBasicAuthHandler.
During verification, the server sends a header (401 error code) request for verification. Scheme and a 'realm' are specified. The format is www-authenticate: SCHEME realm = "realm ".
For example: www-authenticate: Basic realm = "cPanel Users"

The client must use a new request and contain the correct name and password in the request header.
To simplify this process, we can create an HTTPBasicAuthHandler instance and let opener use this handler.
HTTPBasicAuthHandler uses a password management object to process URLs and realms to map user names and passwords.
If you know what realm is, you can use HTTPPasswordMgr.
If you do not care about what realm is, you can use HTTPPasswordMgrWithDefaultRealm.

The Code is as follows:

[Python]View plaincopy

1. #-*-Coding: UTF-8 -*-
2. Import urllib2
4. # Create a password management
5. Password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm ()
6. # Add a user name and password
8. Top_level_url = "http://example.com/foo"
9. Password_mgr.add_password (None, top_level_url, 'why', '123 ')
11. # Create a new handler
12. Handler = urllib2.HTTPBasicAuthHandler (password_mgr)
14. # Create an "opener" (OpenerDirector instance)
15. Opener = urllib2.build _ opener (handler)
17. A_url = 'HTTP: // www.baidu.com /'
19. # Use opener to obtain a URL
20. Opener. open (a_url)
22. # Install opener.
23. # Now all calls to urllib2.urlopen will use our opener.
24. Urllib2.install _ opener (opener)

Note: In addition to HHTPBasicAuthHandler, ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, and HTTPErrorProcessor all return Handler.

Welcome to my public account

Zookeeper