URLLIB2 Custom Opener Detailed _python

The Urllib2.urlopen () function does not support authentication, cookies, or other HTTP advanced features. To support these features, you must use the Build_opener () function to create a custom opener object.

Copy Code code as follows:

Build_opener ([Handler1 [Handler2, ...]])

Parameter handler is a handler example, commonly used in Httpbasicauthhandler, Httpcookieprocessor, Proxyhandler and so on.

The object returned by Build_opener () has the open () method, which is the same as the function of the Urlopen () function.

If you want to modify the HTTP header, you can use:

Copy Code code as follows:

Import Urllib2
Opener = Urllib2.build_opener ()
Opener.addheaders = [(' User-agent ', ' mozilla/5.0 ')]
Opener.open (' http://www.example.com/')

2. Install_opener (opener)

Installs a different opener object as the global opener used by Urlopen ().

3. Password verification (Httpbasicauthhandler)

The Httpbasicauthhandler () handler can use Add_password () to set the password.

Copy Code code as follows:

H.add_password (REALM,URI,USER,PASSWD)

Realm is the name or descriptive information associated with validation, depending on the remote server. The URI is the base URL. User and passwd specify the username and password, respectively.

Copy Code code as follows:

Import Urllib2
Auth=urllib2. Httpbasicauthhandler ()
Auth.add_password (' Administrator ', ' http://www.example.com ', ' Dave ', ' 123456 ')
Opener=urllib2.build_opener (auth)
U=opener.open (' http://www.example.com/evilplan.html ')

4. Cookie processing (httpcookieprocessor)

Copy Code code as follows:

Import Urllib2,cookielib
Cookie=cookielib. Cookiejar ()
Cookiehand=urllib2. Httpcookieprocessor (Cookie)
Opener=urllib2.build_opener (Cookiehand)

5. Agent (Proxyhandler)

The Proxyhandler (proxies) parameter proxies is a dictionary that maps the protocol name (HTTP,FTP) to the URL of the appropriate proxy server.

Copy Code code as follows:

Proxy=proxyhandler ({' http ': ' http://someproxy.com:8080 '})
Auth=httpbasicauthhandler ()
Auth.add_password ()
Opener=build_opener (Auth,proxy)

You can also use proxies in Urlopen

Copy Code code as follows:

Import Urllib2
Proxy = ' http://%s:%s@%s '% (' userName ', ' Password ', ' proxy ')
Information = Urllib2.urlopen ("http://www.example.com", proxies={' http ':p Roxy})