Python33urllib2 usage details

There are many practical tool classes in the Python standard library. here we will summarize the usage details of urllib2: Proxy setting, Timeout setting, adding specific, Cookie to HTTPRequest, using http put and DELETE methods. Proxy settings

By default, urllib2 uses the environment variable http_proxy to set HTTP Proxy. If you want to explicitly control the Proxy in the program without being affected by environment variables, you can use the following method:

The code is as follows:

Import urllib2

Enable_proxy = True
Proxy_handler = urllib2.ProxyHandler ({"http": 'http: // some-proxy.com: 100 '})
Null_proxy_handler = urllib2.ProxyHandler ({})

If enable_proxy:
Opener = urllib2.build _ opener (proxy_handler)
Else:
Opener = urllib2.build _ opener (null_proxy_handler)

Urllib2.install _ opener (opener)

Note that using urllib2.install _ opener () sets the global opener of urllib2. In this way, the subsequent use is very convenient, but it cannot be more fine-grained. for example, you want to use two different Proxy settings in the program. A better way is to directly call opener's open method instead of the global urlopen method.

Timeout settings

In the old version of Python, the urllib2 API does not expose the Timeout settings. to set the Timeout value, you can only change the global Timeout value of the Socket.

The code is as follows:

Import urllib2
Import socket

Socket. setdefatimetimeout (10) # timeout after 10 seconds
Urllib2.socket. setdefatimetimeout (10) # Another method

After Python 2.6, timeout can be directly set through the timeout parameter of urllib2.urlopen.

The code is as follows:

Import urllib2
Response = urllib2.urlopen ('http: // www.google.com ', timeout = 10)

Add a specific Header to the HTTP Request

To add a header, you must use the Request object:

The code is as follows:

Import urllib2

Request = urllib2.Request (uri)
Request. add_header ('User-agent', 'fake-Client ')
Response = urllib2.urlopen (request)

Pay special attention to some headers. the server will check these headers.

User-Agent: some servers or proxies use this value to determine whether the request is sent by the browser.

Content-Type: when the REST interface is used, the server checks the value to determine how to parse the Content in the HTTP Body. Common values include:

Application/xml: used in xml rpc, such as RESTful/SOAP calls
Application/json: used for json rpc calls
Application/x-www-form-urlencoded: Used when the browser submits a Web form
When using the RESTful or SOAP service provided by the server, the Content-Type setting error may cause the server to reject the service.

Redirect

By default, urllib2 automatically performs a redirect action on the HTTP 3XX return code, without manual configuration. To check whether a redirect action has occurred, you only need to check whether the Response URL and Request URL are consistent.

The code is as follows:

Import urllib2
Response = urllib2.urlopen ('http: // www.google.cn ')
Redirected = response. geturl () = 'http: // www.google.cn'

If you do not want automatic redirect, you can customize the HTTPRedirectHandler class in addition to using the lower-level httplib Library.

The code is as follows:

Import urllib2

Class RedirectHandler (urllib2.HTTPRedirectHandler ):
Def http_error_301 (self, req, fp, code, msg, headers ):
Pass
Def http_error_302 (self, req, fp, code, msg, headers ):
Pass

Opener = urllib2.build _ opener (RedirectHandler)
Opener. open ('http: // www.google.cn ')

Cookie

Urllib2 automatically processes cookies. To obtain the value of a Cookie, you can do this:

The code is as follows:

Import urllib2
Import cookielib

Cookie = cookielib. CookieJar ()
Opener = urllib2.build _ opener (urllib2.HTTPCookieProcessor (cookie ))
Response = opener. open ('http: // www.google.com ')
For item in cookie:
If item. name = 'some _ cookie_item_name ':
Print item. value

Use the PUT and DELETE methods of HTTP

Urllib2 only supports http get and POST methods. to use http put and DELETE methods, you can only use httplib libraries of lower layers. Even so, we can make urllib2 send a PUT or DELETE request in the following way:

The code is as follows:

Import urllib2

Request = urllib2.Request (uri, data = data)
Request. get_method = lambda: 'put' # or 'delete'
Response = urllib2.urlopen (request)

Although this method belongs to the Hack method, it is no problem in actual use.

Get the HTTP return code

For 200 OK, you only need to use the getcode () method of the response object returned by urlopen to obtain the HTTP return code. However, for other return codes, urlopen throws an exception. At this time, we need to check the code attribute of the exception object:

The code is as follows:

Import urllib2
Try:
Response = urllib2.urlopen ('http: // www. jb51.ent ')
Failed T urllib2.HTTPError, e:
Print e. code
Debug Log

When using urllib2, you can use the following method to open the debug Log, so that the content of the packet sending and receiving will be printed on the screen for debugging convenience. sometimes you can save the packet capture work.

The code is as follows:

Import urllib2

HttpHandler = urllib2.HTTPHandler (debuglevel = 1)
HttpsHandler = urllib2.HTTPSHandler (debuglevel = 1)
Opener = urllib2.build _ opener (httpHandler, httpsHandler)

Urllib2.install _ opener (opener)
Response = urllib2.urlopen ('http: // www.google.com ')