Python Requests Quick Start introduction

This article mainly introduces the Python requests using the Quick Start tutorial, using requests to send network requests is very simple, how to do, please refer to this article

Get started quickly

Can't wait for it? This page provides a good guide on how to get started with requests. It is assumed that you have installed the requests. If not, go to install a section to see it.

First, make sure that:

Requests already installed

Requests is the newest

Let's start with some simple examples.

Send Request

Sending network requests using requests is straightforward.

To import the requests module at the outset:

>>> Import Requests

Then, try to get a webpage. In this example, we'll get the public timeline for Github:

>>> r = requests.get (' Https://github.com/timeline.json ')

Now, we have a Response object named R. We can get all the information we want from this object.

Requests's simple API means that all HTTP request types are obvious. For example, you can send an HTTP POST request like this:

>>> r = requests.post (http://httpbin.org/post)

Pretty, isn't it? So what about the other HTTP request types: Put,delete,head and OPTIONS? Are the same simple:

>>> r = requests.put ("Http://httpbin.org/put") >>> r = Requests.delete ("Http://httpbin.org/delete") >>> r = Requests.head ("Http://httpbin.org/get") >>> r = requests.options (http://httpbin.org/get)

All very well, but this is only the tip of the iceberg of requests.

Passing URL parameters

You might often want to pass some kind of data to the query string of a URL. If you build the URL manually, the data is placed in the URL in the form of a key/value pair followed by a question mark. For example, Httpbin.org/get?key=val. Requests allows you to use the params keyword parameter to provide these parameters as a string dictionary. For example, if you want to pass key1=value1 and key2=value2 to Httpbin.org/get, then you can use the following code:

>>> payload = {' Key1 ': ' value1 ', ' key2 ': ' value2 '}>>> r = Requests.get ("Http://httpbin.org/get", Params=payload)

By printing out the URL, you can see that the URL has been correctly encoded:

>>> print (R.url) http://httpbin.org/get?key2=value2&key1=value1

Note that none of the keys in the dictionary will be added to the URL's query string.

You can also pass in a list as a value:

>>> payload = {' Key1 ': ' value1 ', ' key2 ': [' value2 ', ' value3 ']}>>> r = Requests.get (' http://httpbin.org /get ', params=payload) >>> print (R.url) http://httpbin.org/get?key1=value1&key2=value2&key2= Value3

Response Content

We can read the contents of the server response. Again, take the GitHub timeline as an example:

>>> Import requests>>> r = Requests.get (' Https://github.com/timeline.json ') >>> r.textu ' [{ "Repository": {"open_issues": 0, "url": "https://github.com/...

Requests will automatically decode the content from the server. Most Unicode character sets can be decoded seamlessly.

After the request is issued, requests will make an educated guess based on the HTTP header's encoding of the response. When you visit R.text, requests uses its inferred text encoding. You can find out what encoding requests is using and can use the R.encoding property to change it:

>>> r.encoding ' utf-8 ' >>> r.encoding = ' iso-8859-1 '

If you change the code, the Request will use the new value of r.encoding whenever you visit r.text. You may want to modify the encoding by using special logic to calculate the encoding of the text. For example, HTTP and XML itself can specify the encoding. In this case, you should use R.content to find the encoding, and then set the r.encoding to the corresponding encoding. This makes it possible to parse the r.text using the correct encoding.

Requests can also use custom encodings where you need them. If you create your own code and register with the codecs module, you can easily use the decoder name as the r.encoding value, and then the requests will handle the encoding for you.

Binary response Content

You can also access the request response body in bytes, for non-textual requests:

>>> R.contentb ' [{"repository": {"open_issues": 0, "url": "https://github.com/...

Requests will automatically decode your gzip and deflate transmit encoded response data for you.

For example, to create a picture of the binary data returned by the request, you can use the following code:

>>> from PIL import image>>> from io import bytesio>>> i = Image.open (Bytesio (r.content))

JSON Response Content

The requests also has a built-in JSON decoder to help you work with JSON data:

>>> Import requests>>> r = Requests.get (' Https://github.com/timeline.json ') >>> R.json () [{ U ' repository ': {u ' open_issues ': 0, U ' url ': ' https://github.com/...

If the JSON decoding fails, R.json () throws an exception. For example, the response content is 401 (unauthorized), and attempting to access R.json () will throw Valueerror:no JSON object could be decoded exception.

It is important to note that the successful invocation of R.json () and * * * does not imply a successful response. Some servers include a JSON object (such as the error details of HTTP 500) in a failed response. This JSON will be decoded back. To check whether the request was successful, use R.raise_for_status () or check to see if the R.status_code is the same as your expectations.

Original response Content

In rare cases, you may want to get the original socket response from the server, then you can access the R.raw. If you really want to do this, make sure that you set the stream=true in the initial request. Specifically you can do this:

>>> r = requests.get (' Https://github.com/timeline.json ', stream=true) >>> r.raw< Requests.packages.urllib3.response.HTTPResponse object at 0x101194810>>>> R.raw.read (Ten) ' \x1f\x8b\x08\ X00\x00\x00\x00\x00\x00\x03 '

In general, however, you should save the text stream to a file in the following mode:

with open (filename, ' WB ') as Fd:for Chunk in R.iter_content (chunk_size):  fd.write (Chunk)

Using response.iter_content will handle a lot of what you have to deal with directly using Response.raw. When streaming, the above is the preferred way to get content. Note that chunk_size can is freely adjusted to a number which may better fit your use cases.

Custom Request Headers

If you want to add an HTTP header for the request, simply pass a dict to the headers parameter.

For example, in the previous example we did not specify Content-type:

>>> url = ' Https://api.github.com/some/endpoint '
>>> headers = {' user-agent ': ' my-app/0.0.1 '}
>>> r = requests.get (URL, headers=headers)

Note: Custom headers have a lower priority than some specific sources of information, such as:

If the user authentication information is set in the. NETRC, the authorization that is set with the headers= will not take effect. If the auth= parameter is set, the ". Netrc" setting is invalid.

If it is redirected to another host, the authorization header is removed.

The proxy authorization header is overwritten by the proxy identity provided in the URL.

When we can judge the length of the content, the content-length of the header will be rewritten.

Further, requests does not change its behavior based on the specifics of the custom header. Only in the final request, all header information will be passed in.

Note: All header values must be string, bytestring, or Unicode. Although it is permissible to pass the Unicode header, it is not recommended.

More complex POST Requests

Typically, you want to send some data encoded as forms--much like an HTML form. To implement this, simply pass a dictionary to the data parameter. Your data dictionary is automatically encoded as a form when the request is made:

>>> payload = {' Key1 ': ' value1 ', ' key2 ': ' value2 '}

>>> r = requests.post ("Http://httpbin.org/post", Data=payload)
>>> Print (R.text)
{
...
"Form": {
"Key2": "value2",
"Key1": "Value1"
},
...
}

You can also pass in a tuple list for the data parameter. This is especially effective when multiple elements in a form use the same key:

>>> payload = ((' Key1 ', ' value1 '), (' Key1 ', ' value2 '))
>>> r = requests.post (' http://httpbin.org/post ', data=payload)
>>> Print (R.text)
{
...
"Form": {
"Key1": [
"Value1",
"Value2"
]
},
...
}

Many times the data you want to send is not encoded as a form. If you pass a string instead of a dict, the data will be published directly.

For example, the Github API V3 accepts Post/patch data encoded as JSON:

>>> Import JSON
>>> url = ' Https://api.github.com/some/endpoint '
>>> payload = {' Some ': ' Data '}
>>> r = requests.post (URL, data=json.dumps (payload))

In addition to the ability to encode dict yourself, you can also use JSON parameters to pass directly, and then it will be automatically encoded. This is a new addition to the 2.4.2 version:

>>> url = ' Https://api.github.com/some/endpoint '
>>> payload = {' Some ': ' Data '}
>>> r = requests.post (URL, json=payload)

Post a multi-part encoding (multipart-encoded) file

Requests makes it easy to upload multi-part encoded files:

>>> url = ' Http://httpbin.org/post '
>>> files = {' file ': Open (' Report.xls ', ' RB ')}
>>> r = requests.post (URL, files=files)
>>> R.text
{
...
"Files": {
"File": "<censored...binary...data>"
},
...
}

You can explicitly set the filename, file type, and request header:

>>> url = ' Http://httpbin.org/post '
>>> files = {' file ': (' Report.xls ', open (' Report.xls ', ' RB '), ' application/vnd.ms-excel ', {' Expires ': ' 0 '})}
>>> r = requests.post (URL, files=files)
>>> R.text
{
...
"Files": {
"File": "<censored...binary...data>"
},
...
}

If you want, you can also send a string as a file to receive:

>>> url = ' Http://httpbin.org/post '
>>> files = {' file ': (' report.csv ', ' some,data,to,send\nanother,row,to,send\n ')}

>>> r = requests.post (URL, files=files)
>>> R.text
{
...
"Files": {
"File": "some,data,to,send\\nanother,row,to,send\\n"
},
...
}

If you send a very large file as a multipart/form-data request, you may want to make the request into a data stream. Requests is not supported by default, but a third-party package Requests-toolbelt is supported. You can read the Toolbelt documentation to learn how to use it.

Send a multi-file reference in one request to the Advanced Usage section.

Warning

We strongly recommend that you open the file in binary mode. This is because requests may try to provide you with the Content-length header, which is set to the number of bytes (bytes) of the file when it does so. If you open the file in text mode, an error may occur.

Response Status Code

We can detect the response status code:

>>> r = requests.get (' Http://httpbin.org/get ')
>>> R.status_code
200

For easy reference, requests also comes with a built-in status code query object:

>>> R.status_code = = Requests.codes.ok
True

If you send an error request (a 4XX client error, or a 5XX server error response), we can throw an exception by Response.raise_for_status ():

>>> Bad_r = requests.get (' http://httpbin.org/status/404 ')
>>> Bad_r.status_code
404
>>> Bad_r.raise_for_status ()
Traceback (most recent):
File "requests/models.py", line 832, in Raise_for_status
Raise Http_error
requests.exceptions.httperror:404 Client Error

However, since the Status_code of R in our example is 200, when we call Raise_for_status (), we get:

>>> R.raise_for_status ()
None

Everything is very harmonious, huh.

Response header

We can view the server response headers shown in a Python dictionary:

>>> r.headers
{
' content-encoding ': ' gzip ',
' transfer-encoding ': ' chunked ',
' Connection ': ' Close ',
' Server ': ' nginx/1.0.4 ',
' X-runtime ': ' 148ms ',
' ETag ': ' "e1ca502697e5c9317743dc078f67693f" ',
' Content-type ': ' Application/json '
}

But this dictionary is very special: it is only for HTTP headers. According to RFC 2616, the HTTP header is case insensitive.

Therefore, we can use any uppercase form to access these response header fields:

>>> r.headers[' Content-type ']
' Application/json '
>>> r.headers.get (' Content-type ')
' Application/json '

It also has a special point that the server can accept the same header multiple times, using different values each time. But requests will merge them so that they can be represented by a map, see RFC 7230:

A recipient may combine multiple header fields with the same field name into one "Field-name:field-value" pair, without C Hanging the semantics of the message, by appending all subsequent field value to the combined field value in order, Separ Ated by a comma.

The receiver can merge multiple header fields of the same name, combine them into a "field-name:field-value" pair, append each subsequent field value to the combined field value, separated by commas, which does not change the semantics of the information.

Cookies

If a response contains some cookies, you can quickly access them:

>>> url = ' Http://example.com/some/cookie/setting/url '
>>> r = requests.get (URL)
>>> r.cookies[' Example_cookie_name ']
' Example_cookie_value '
To send your cookies to the server, you can use the Cookies parameter:

>>> url = ' http://httpbin.org/cookies '
>>> cookies = dict (cookies_are= ' working ')

>>> r = requests.get (URL, cookies=cookies)
>>> R.text
' {' "cookies": {"Cookies_are": "Working"}} '

The return object of the Cookie is Requestscookiejar, which behaves like a dictionary, but with a more complete interface, which is suitable for cross-path use across domains. You can also upload the Cookie Jar to requests:

>>> jar = Requests.cookies.RequestsCookieJar () >>> jar.set (' Tasty_cookie ', ' yum ', domain= ' Httpbin.org ', path= '/cookies ') >>> jar.set (' Gross_cookie ', ' Blech ', domain= ' httpbin.org ', path= '/elsewhere ' ) >>> url = ' http://httpbin.org/cookies ' >>> r = requests.get (URL, cookies=jar) >>> r.text ' {" Cookies ": {" Tasty_cookie ":" Yum "}} '

Redirect and request history

By default, requests automatically handles all redirects except HEAD.

You can use the history method of the response object to track the redirection.

Response.history is a list of Response objects that were created to complete the request. This list of objects is sorted by the most recent request.

For example, Github redirects all HTTP requests to HTTPS:

>>> r = requests.get (' http://github.com ') >>> r.url ' https://github.com/' >>> r.status_ Code200>>> r.history[<response [301]>]

If you are using GET, OPTIONS, POST, PUT, PATCH, or DELETE, you can disable redirection with the Allow_redirects parameter:

>>> r = requests.get (' http://github.com ', allow_redirects=false) >>> r.status_code301>>> R.history[]

If you use HEAD, you can also enable redirection:

>>> r = requests.head (' http://github.com ', allow_redirects=true) >>> r.url ' https://github.com/' >>> r.history[<response [301]>]

Timeout

You can tell requests to stop waiting for a response after the number of seconds set by the timeout parameter. Basically all production code should use this parameter. If you do not use it, your program may lose its response forever:

>>> requests.get (' http://github.com ', timeout=0.001) Traceback (most recent call last): File "<stdin>", Line 1, in <module>requests.exceptions.timeout:httpconnectionpool (host= ' github.com ', port=80): Request timed Out. (timeout=0.001)

Attention

Timeout is only valid for the connection process and is not related to the download of the response body. Timeout is not a time limit for the entire download response, but if the server does not answer within timeout seconds, an exception will be thrown (more precisely, when no byte data is received from the underlying socket in timeout seconds) if no timeout is Specified explicitly, requests does not time out.

Errors and exceptions

Requests throws a Connectionerror exception when encountering network problems such as DNS query failure, connection rejection, and so on.

If an HTTP request returns an unsuccessful status code, Response.raise_for_status () throws a Httperror exception.

If the request times out, a timeout exception is thrown.

If the request exceeds the maximum number of redirects set, a Toomanyredirects exception is thrown.

All exceptions that are explicitly thrown by requests inherit from Requests.exceptions.RequestException.