The module used to process HTTP in the Python standard library is URLLIB2, but the API is fragmented, and requests is a simpler and more user-friendly third-party library.
Download with Pip:
PIP Install requests
or git:
git clone git://github.com/kennethreitz/requests.git
Send request:
Get method
>>> Import requests>>> r = Requests.get (' https://api.github.com/events ')
Post method:
>>> r = requests.post ("Http://httpbin.org/post")
You can also use other methods:
>>> r = requests.put ("Http://httpbin.org/put") >>> r = Requests.delete ("Http://httpbin.org/delete") >>> r = Requests.head ("Http://httpbin.org/get") >>> r = requests.options ("Http://httpbin.org/get")
You can also put the request method in the parameter:
>>> Import requests>>> req = requests.request (' GET ', ' http://httpbin.org/get ')
To pass parameters or upload files:
1. If you want to pass parameters in a URL, use the params parameter, which can be a dictionary or a string:
>>> payload = {' Key1 ': ' value1 ', ' key2 ': ' value2 '}>>> r = Requests.get ("Http://httpbin.org/get", params=payload) >>> r.urlu ' http://httpbin.org/get?key2=value2&key1=value1 '
2. If you want to pass the parameter in the request body, use the data parameter, which can be a dictionary, a string, or a class file object.
When using a dictionary, form-encoded data is sent:
>>> payload = {' Key1 ': ' value1 ', ' key2 ': ' value2 '}>>> r = requests.post ("Http://httpbin.org/post", Data=payload) >>> Print (r.text) { ... " Form ": { " Key2 ":" value2 ", " Key1 ":" Value1 " }, ...}
Data is sent directly when using a string:
>>> import json>>> url = ' https://api.github.com/some/endpoint ' >>> payload = {' Some ': ' Data ' }>>> r = requests.post (URL, data=json.dumps (payload))
3. If you want to upload a file, you can send the multipart-encoded data using the files parameter, which is a dictionary of {' name ': file-like-objects} format (or {' name ':(' filename ', fileobj )}) :
>>> url = ' Http://httpbin.org/post ' >>> files = {' file ': Open (' Report.xls ', ' RB ')}>>> r = Reque Sts.post (URL, files=files) >>> r.text{... " Files ": { " file ":" <censored...binary...data> " }, ...}
You can also explicitly set the filename, Content_Type and headers:
>>> url = ' Http://httpbin.org/post ' >>> files = {' file ': (' Report.xls ', open (' Report.xls ', ' RB '), ' Application/vnd.ms-excel ', {' Expires ': ' 0 '})}>>> r = requests.post (URL, files=files) >>> print R.text { "args": {}, "Data": "", "files": { "file": "1\t2\r\n" }, "form": {}, "headers": { "Content-type": "Multipart/form-data; boundary=e0f9ff1303b841498ae53a903f27e565 ", " Host ":" httpbin.org ", " user-agent ":" python-requests/2.2.1 cpython/2.7.3 WINDOWS/7 ", }, " url ":" Http://httpbin.org/post "}
Set headers
>>> import json>>> url = ' https://api.github.com/some/endpoint ' >>> payload = {' Some ': ' Data ' }>>> headers = {' Content-type ': ' Application/json '}>>> r = requests.post (URL, data=json.dumps ( Payload), Headers=headers)
Response object:
Gets the Unicode string, which is automatically decoded according to the character encoding (r.encoding) of the response header, and can, of course, set r.encoding itself:
>>> r = requests.get (' Https://github.com/timeline.json ') >>> r.textu ' {"message": "Hello there, Wayfaring Stranger ...
Gets the bytes string that automatically decodes the gzip and deflate data:
>>> r.content ' {"message": "Hello there, wayfaring stranger ...
To store a Web picture, you can:
>>> from PIL import image>>> from Stringio import stringio>>> i = Image.open (Stringio (R.conten T))
You can decode a JSON object:
>>> R.json () {u ' documentation_url ': U ' https://developer ...
To return the raw response, you need to set the stream to true in the requests request:
>>> r = requests.get (' Https://github.com/timeline.json ', stream=true) >>> r.raw< Requests.packages.urllib3.response.HTTPResponse object at 0x101194810>>>> R.raw.read (Ten) ' \x1f\x8b\x08\ X00\x00\x00\x00\x00\x00\x03 '
Of course, the better approach is to iterate over the data:
with open (filename, ' WB ') as FD: For Chunk in R.iter_content (chunk_size): fd.write (Chunk)
Get the response code:
>>> r = requests.get (' http://httpbin.org/get ') >>> r.status_code200
Get Response headers:
>>> r.headers{' content-encoding ': ' gzip ', ' transfer-encoding ': ' chunked ', ' connection ': ' Close ', ' server ': ' nginx/1.0.4 ', ' x-runtime ': ' 148ms ', ' etag ': ' "e1ca502697e5c9317743dc078f67693f" ', ' content-type ': ' Application/json '}
Cookies
Gets the cookie that returns the Cookiejar object:
>>> url = ' http://www.baidu.com ' >>> r = requests.get (URL) >>> r.cookies
Convert Cookiejar to Dictionary:
>>> Requests.utils.dict_from_cookiejar (r.cookies) {' Baiduid ': ' 84722199df8edc372d549ec56ca1a0e2:fg=1 ', ' Bd_home ': ' 0 ', ' bdsvrtm ': ' 0 '}
To convert a dictionary to Cookiejar:
Requests.utils.cookiejar_from_dict (Cookie_dict, Cookiejar=none, Overwrite=true)
Uploading your own set of cookies, using cookies, can be a dictionary or Cookiejar object:
>>> url = ' http://httpbin.org/cookies ' >>> cookies = dict (cookies_are= ' working ') >>> r = Requests.get (URL, cookies=cookies) >>> r.text ' {"cookies": {"Cookies_are": "Working"}} '
If you need to keep cookies in your session, you need to use the following session.
Redirection and history
You can use the history property to track redirection
>>> r = requests.get (' http://github.com ') >>> r.url ' https://github.com/' >>> r.status_ Code200>>> r.history[<response [301]>]
Session
To preserve state in a session, you can use request. Session ().
Session can use Get,post, etc., the returned cookie will be automatically retained on the next visit:
>>> Import requests>>> s = requests. Session () >>> s.get (' Http://httpbin.org/get ') 200
You can also set headers,cookies yourself:
>>> Import requests>>> s = requests. Session () >>> s.headers={...}
Verify
Basic Authentication
>>> from Requests.auth import httpbasicauth>>> requests.get (' Https://api.github.com/user ', auth= Httpbasicauth (' user ', ' pass ') <response [200]>
Because HTTP Basic auth is common, you can also verify it directly:
>>> requests.get (' Https://api.github.com/user ', auth= (' user ', ' Pass ')) <response [200]>
Digest Authentication
>>> from requests.auth import httpdigestauth>>> url = ' http://httpbin.org/digest-auth/auth/user/ Pass ' >>> requests.get (URL, auth=httpdigestauth (' user ', ' Pass ')) <response [200]>
OAuth 1 Authentication
>>> Import requests>>> from requests_oauthlib import oauth1>>> url = ' https:// Api.twitter.com/1.1/account/verify_credentials.json ' >>> auth = OAuth1 (' Your_app_key ', ' Your_app_secret ', ' User_oauth_token ', ' User_oauth_token_secret ') >>> requests.get (URL, auth=auth) <response [200]>
You can also use validation classes that you write yourself. For example, a Web service accepts authentication that sets the X-pizza header to a password, which can be written to verify the class:
From Requests.auth import Authbaseclass Pizzaauth (authbase): "" "attaches HTTP Pizza authentication to the given Reque St object. "" " def __init__ (self, username): # Setup No auth-related data here Self.username = Username def __call__ (self, R): # Modify and return the request r.headers[' X-pizza '] = Self.username return r
Use:
>>> requests.get (' Http://pizzabin.org/admin ', Auth=pizzaauth (' Kenneth ')) <response [200]>
Agent
Using proxies:
Import requestsproxies = { "http": "http://10.10.1.10:3128", "https": "http://10.10.1.10:1080",} Requests.get ("http://example.org", proxies=proxies)
You can set environment variables:
$ export http_proxy= "http://10.10.1.10:3128" $ export https_proxy= "http://10.10.1.10:1080" $ python>>> Import Requests>>> requests.get ("http://example.org")
If the agent needs validation:
Proxies = { "http": "Http://user:[email protected]:3128/",}
Python Requests Library: HTTP for humans