Detailed description of urllib2 custom opener

The urllib2.urlopen () function does not support authentication, cookie, or other advanced HTTP functions. To support these functions, you must use the build_opener () function to create a custom Opener object.

Copy codeThe Code is as follows:
Build_opener ([handler1 [handler2,...])

Handler is a Handler instance. HTTPBasicAuthHandler, HTTPCookieProcessor, and ProxyHandler are commonly used.

The objects returned by build_opener () have the open () method, which is the same as the functions of the urlopen () function.

To modify the http header, you can use:

Copy codeThe Code is as follows:
Import urllib2
Opener = urllib2.build _ opener ()
Opener. addheaders = [('user-agent', 'mozilla/5.0 ')]
Opener. open ('HTTP: // www.example.com /')

2. install_opener (opener)

Install different opener objects as the global opener used by urlopen.

3. Password verification (HTTPBasicAuthHandler)

The HTTPBasicAuthHandler () handler can use add_password () to set the password.

Copy codeThe Code is as follows:
H. add_password (realm, uri, user, passwd)

Realm is the name or description associated with the verification, depending on the remote server. Uri is the base URL. User and passwd specify the user name and password respectively.

Copy codeThe Code is as follows:
Import urllib2
Auth = urllib2.HTTPBasicAuthHandler ()
Auth. add_password ('admin', 'HTTP: // www.example.com ', 'Dave', '123 ')
Opener = urllib2.build _ opener (auth)
U = opener. open ('HTTP: // www.example.com/evilplan.html ')

4. Cookie processing (HTTPCookieProcessor)

Copy codeThe Code is as follows:
Import urllib2, cookielib
Cookie = cookielib. CookieJar ()
Cookiehand = urllib2.HTTPCookieProcessor (cookie)
Opener = urllib2.build _ opener (cookiehand)

5. Proxy (ProxyHandler)

The ProxyHandler (proxies) parameter proxies is a dictionary that maps protocol names (http, ftp) and so on to the URL of the corresponding proxy server.

Copy codeThe Code is as follows:
Proxy = ProxyHandler ({'http': 'http: // someproxy.com: 8080 '})
Auth = HTTPBasicAuthHandler ()
Auth. add_password ()
Opener = build_opener (auth, proxy)

You can also use a proxy in urlopen.

Copy codeThe Code is as follows:
Import urllib2
Proxy = 'HTTP: // % s: % s @ % s' % ('username', 'Password', 'proxy ')
InforMation = urllib2.urlopen ("http://www.example.com", proxies = {'HTTP ': proxy })