Python Requests Library: HTTP for Humans, pythonrequests

Python Requests Library: HTTP for Humans, pythonrequests

In the Python standard library, urllib2 is used to process HTTP. However, the API is too fragmented, and requests is a simple and user-friendly third-party library.

Download with pip:

pip install requests

Or git:

git clone git://github.com/kennethreitz/requests.git

Send request:

GET Method

>>> import requests>>> r = requests.get('https://api.github.com/events')

POST method:

>>> r = requests.post("http://httpbin.org/post")

You can also use other methods:

>>> r = requests.put("http://httpbin.org/put")>>> r = requests.delete("http://httpbin.org/delete")>>> r = requests.head("http://httpbin.org/get")>>> r = requests.options("http://httpbin.org/get")

You can also put the request method in the parameter:

>>> import requests>>> req = requests.request('GET', 'http://httpbin.org/get')

Passing parameters or uploading files:

1. If you want to put parameters in the url for transmission, use the params parameter, which can be a dictionary or string:

>>> payload = {'key1': 'value1', 'key2': 'value2'}>>> r = requests.get("http://httpbin.org/get", params=payload)>>> r.urlu'http://httpbin.org/get?key2=value2&key1=value1'

2. If you want to pass parameters in the request body, you can use the data parameter, which can be a dictionary, string, or a file-like object.

When the dictionary is used, form-encoded data will be sent:

>>> payload = {'key1': 'value1', 'key2': 'value2'}>>> r = requests.post("http://httpbin.org/post", data=payload)>>> print(r.text){  ...  "form": {    "key2": "value2",    "key1": "value1"  },  ...}

Data is directly sent when strings are used:

>>> import json>>> url = 'https://api.github.com/some/endpoint'>>> payload = {'some': 'data'}>>> r = requests.post(url, data=json.dumps(payload))

Stream upload:

with open('massive-body', 'rb') as f:    requests.post('http://some.url/streamed', data=f)

Chunk-Encoded upload:

def gen():    yield 'hi'    yield 'there'requests.post('http://some.url/chunked', data=gen())

3. to upload a file, you can use the file parameter to send Multipart-encoded data. The file parameter is {'name ': dictionary in file-like-objects} format (or {'name' :( 'filename', fileobj )}):

>>> url = 'http://httpbin.org/post'>>> files = {'file': open('report.xls', 'rb')}>>> r = requests.post(url, files=files)>>> r.text{  ...  "files": {    "file": "<censored...binary...data>"  },  ...}

You can also explicitly set filename, content_type and headers:

>>> url = 'http://httpbin.org/post'>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}>>> r = requests.post(url, files=files)>>> print r.text{  "args": {},   "data": "",   "files": {    "file": "1\t2\r\n"  },   "form": {},   "headers": {    "Content-Type": "multipart/form-data; boundary=e0f9ff1303b841498ae53a903f27e565",     "Host": "httpbin.org",     "User-Agent": "python-requests/2.2.1 CPython/2.7.3 Windows/7",   },   "url": "http://httpbin.org/post"}

Upload multiple files at a time:

>>> url = 'http://httpbin.org/post'>>> multiple_files = [('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),                      ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]>>> r = requests.post(url, files=multiple_files)>>> r.text{  ...  'files': {'images': 'data:image/png;base64,iVBORw ....'}  'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7aa0b3a',  ...}

Set Headers

>>> import json>>> url = 'https://api.github.com/some/endpoint'>>> payload = {'some': 'data'}>>> headers = {'content-type': 'application/json'}>>> r = requests.post(url, data=json.dumps(payload), headers=headers)

Response object:

After obtaining the unicode string, it will automatically perform Decoding Based on the character encoding (r. encoding) in the Response Header. Of course, you can also set r. encoding:

>>> r = requests.get('https://github.com/timeline.json')>>> r.textu'{"message":"Hello there, wayfaring stranger...

After obtaining the bytes string, gzip and deflate data are automatically decoded:

>>> r.content'{"message":"Hello there, wayfaring stranger. ..

To store web images, you can:

>>> from PIL import Image>>> from StringIO import StringIO>>> i = Image.open(StringIO(r.content))

Json object decoding:

>>> r.json(){u'documentation_url': u'https://developer...

Return raw response. Set stream to True in the requests request:

>>> r = requests.get('https://github.com/timeline.json', stream=True)>>> r.raw<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>>>> r.raw.read(10)'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

If you do not want to process all the data at once, you can:

tarball_url = 'https://github.com/kennethreitz/requests/tarball/master'r = requests.get(tarball_url, stream=True)if int(r.headers['content-length']) < TOO_LONG:  content = r.content  ...

You can also process data iteratively:

with open(filename, 'wb') as fd:    for chunk in r.iter_content(chunk_size):        fd.write(chunk)

Or:

import jsonimport requestsr = requests.get('http://httpbin.org/stream/20', stream=True)for line in r.iter_lines():    # filter out keep-alive new lines    if line:        print(json.loads(line))

Get response code:

>>> r = requests.get('http://httpbin.org/get')>>> r.status_code200

Get Response headers:

>>> r.headers{    'content-encoding': 'gzip',    'transfer-encoding': 'chunked',    'connection': 'close',    'server': 'nginx/1.0.4',    'x-runtime': '148ms',    'etag': '"e1ca502697e5c9317743dc078f67693f"',    'content-type': 'application/json'}

Get sent headers

>>> r.request.headers{'Accept-Encoding': 'identity, deflate, compress, gzip','Accept': '*/*', 'User-Agent': 'python-requests/1.2.0'}

Cookie

Obtain the cookie and return the CookieJar object:

>>> url = 'http://www.baidu.com'>>> r = requests.get(url)>>> r.cookies

Convert CookieJar into a dictionary:

>>> requests.utils.dict_from_cookiejar(r.cookies){'BAIDUID': '84722199DF8EDC372D549EC56CA1A0E2:FG=1', 'BD_HOME': '0', 'BDSVRTM': '0'}

Convert the dictionary into CookieJar:

requests.utils.cookiejar_from_dict(cookie_dict, cookiejar=None, overwrite=True)

Upload the cookie set by yourself. The cookie parameter can be a dictionary or CookieJar object:

>>> url = 'http://httpbin.org/cookies'>>> cookies = dict(cookies_are='working')>>> r = requests.get(url, cookies=cookies)>>> r.text'{"cookies": {"cookies_are": "working"}}'

If you need to retain the cookie in the Session, you need to use the Session to be mentioned later.

Redirection and History

You can use the history attribute to track redirection.

>>> r = requests.get('http://github.com')>>> r.url'https://github.com/'>>> r.status_code200>>> r.history[<Response [301]>]

Session

To retain the Session status, you can use request. Session ().

The Session can use get, post, and other methods. The Session object allows you to retain certain parameters and automatically set cookies during the request.

S = requests. session () s. get ('HTTP: // httpbin.org/cookies/set/sessioncookie/123456789') # Keep cookie in s r = s. get ("http://httpbin.org/cookies") # cookieprint (r. text) # '{"cookies": {"sessioncookie": "123456789 "}}'

You can also set headers and cookies by yourself:

S = requests. session () s. auth = ('user', 'pass') s. headers. update ({'x-test': 'true'}) s. get ('HTTP: // httpbin.org/headers', headers = {'x-test2': 'true'}) # 'x-test' and 'x-test2' will be sent

Preset Request

You can make additional settings before sending a request.

from requests import Request, Sessions = Session()req = Request('GET', url,    data=data,    headers=header)prepped = req.prepare()# do something with prepped.body# do something with prepped.headersresp = s.send(prepped,    stream=stream,    verify=verify,    proxies=proxies,    cert=cert,    timeout=timeout)print(resp.status_code)　

Verify

Basic Authentication

>>> from requests.auth import HTTPBasicAuth>>> requests.get('https://api.github.com/user', auth=HTTPBasicAuth('user', 'pass'))<Response [200]>

Because HTTP Basic Auth is very common, you can also directly verify it:

>>> requests.get('https://api.github.com/user', auth=('user', 'pass'))<Response [200]>

Digest Authentication

>>> from requests.auth import HTTPDigestAuth>>> url = 'http://httpbin.org/digest-auth/auth/user/pass'>>> requests.get(url, auth=HTTPDigestAuth('user', 'pass'))<Response [200]>

OAuth 1 Authentication

>>> import requests>>> from requests_oauthlib import OAuth1>>> url = 'https://api.twitter.com/1.1/account/verify_credentials.json'>>> auth = OAuth1('YOUR_APP_KEY', 'YOUR_APP_SECRET',                  'USER_OAUTH_TOKEN', 'USER_OAUTH_TOKEN_SECRET')>>> requests.get(url, auth=auth)<Response [200]>

You can also use your own verification class. For example, if a web Service accepts password verification by setting the X-Pizza header, you can write the verification class as follows:

from requests.auth import AuthBaseclass PizzaAuth(AuthBase):    """Attaches HTTP Pizza Authentication to the given Request object."""    def __init__(self, username):        # setup any auth-related data here        self.username = username    def __call__(self, r):        # modify and return the request        r.headers['X-Pizza'] = self.username        return r

Usage:

>>> requests.get('http://pizzabin.org/admin', auth=PizzaAuth('kenneth'))<Response [200]>

SSL certificate verification

Check the ssl Certificate of the Host:

>>> requests.get('https://kennethreitz.com', verify=True)    raise ConnectionError(e)ConnectionError: HTTPSConnectionPool(host='kennethreitz.com', port=443): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 10061] )

Github has:

>>> requests.get('https://github.com', verify=True)<Response [200]>

If you set verification to False, you can also ignore SSL certificate verification.

You can read the verification file:

>>> requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))

Proxy

Use Proxy:

import requestsproxies = {  "http": "http://10.10.1.10:3128",  "https": "http://10.10.1.10:1080",}requests.get("http://example.org", proxies=proxies)

You can set environment variables:

$ export HTTP_PROXY="http://10.10.1.10:3128"$ export HTTPS_PROXY="http://10.10.1.10:1080"$ python>>> import requests>>> requests.get("http://example.org")

If the proxy needs verification:

proxies = {    "http": "http://user:pass@10.10.1.10:3128/",}

　　

 

How to install python requests

Linux in terminal:
Sudo pip install requests
 
Which database does python use to parse http data packets?

Let's take a look at urllib, urllib2, httplib, and httplib2. There are four of them, and you should be able to implement all the functions you need.