Python network programming package use tutorial, pythonrequests
I have long heard that the requests library is powerful, but I haven't touched it yet. Today I have touched on it and found that urllib, urllib2, and other methods used in the past are too tricky ......
Here, we will write some simple operations and use them as a preliminary record.
1. Download
Official project page: https://pypi.python.org/pypi/requests/#downloads
You can download it directly from above.
2. Send a get request without Parameters
>>> r = requests.get('http://httpbin.org/get')>>> print r.text{ "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "close", "Host": "httpbin.org", "User-Agent": "python-requests/2.3.0 CPython/2.6.6 Windows/7", "X-Request-Id": "8a28bbea-55cd-460b-bda3-f3427d66b700" }, "origin": "124.192.129.84", "url": "http://httpbin.org/get"}
3. Send a get request with ParametersPut the key and value in a dictionary and pass the params parameter. The function is equivalent to urllib. urlencode.
>>> Import requests >>> pqyload = {'q': 'yang yanxing' >>> r = requests. get ('HTTP: // www.so.com/s', Params = pqyload) >>> r. urlu 'HTTP: // www.so.com/s? Q = % E6 % 9D % A8 % E5 % BD % A6 % E6 % 98% 9f'
4. Send a post request,Passed through the data parameter,
>>> Payload = {'A': 'yang', 'B': 'hello'} >>> r = requests. post ("http://httpbin.org/post", data = payload)> print r. text {"args" :{}, "data": "", "files" :{}, "form": {"a": "\ u6768 ", "B": "hello"}, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection ": "close", "Content-Length": "19", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org ", "User-Agent": "python-requests/2.3.0 CPython/2.6.6 Windows/7", "X-Request-Id": "c81cb937-04b8-4a2d-ba32-04b5c0b3ba98"}, "json": null, "origin": "124.192.129.84", "url": "http://httpbin.org/post" >>>
As you can see, the post parameter has been passed to form. data not only accepts dictionary-type data, but also supports json and other formats.
>>> Payload = {'A': 'yang', 'B': 'hello'} >>> import json >>> r = requests. post ('HTTP: // httpbin.org/post', data = json. dumps (payload ))
5. The post type of the file to be sent. This is equivalent to uploading an image or document to a website. In this case, the files parameter is used.
>>> url = 'http://httpbin.org/post'>>> files = {'file': open('touxiang.png', 'rb')}>>> r = requests.post(url, files=files)
Custom headers, transmitted using headers Parameters
>>> import json>>> url = 'https://api.github.com/some/endpoint'>>> payload = {'some': 'data'}>>> headers = {'content-type': 'application/json'}>>> r = requests.post(url, data=json.dumps(payload), headers=headers)
Vi. Response content
Response status code:
r = requests.get('http://httpbin.org/get')print r.status_code
Response Header:
>>> print r.headers{'content-length': '519', 'server': 'gunicorn/18.0', 'connection': 'keep-alive', 'date': 'Sun, 15 Jun 2014 14:19:52 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'}
You can also obtain this individual response header for some judgment. The parameters here are case insensitive.
r.headers[‘Content-Type']r.headers.get(‘Content-Type')
Response content, which has already been applied:
r.textr.content
7. obtain cookies in response
>>> r = requests.get('http://www.baidu.com')>>> r.cookies['BAIDUID']'D5810267346AEFB0F25CB0D6D0E043E6:FG=1'
You can also customize COOKIES for requests.
>>> url = 'http://httpbin.org/cookies'>>> cookies = {'cookies_are':'working'}>>> r = requests.get(url,cookies = cookies)>>> >>> print r.text{ "cookies": { "cookies_are": "working" }}>>>
There are still a lot of cookies, because I still don't have a lot of cookies at the moment. let's expand it later.
8. Use the timeout parameter to set the timeout time
>>> requests.get('http://github.com', timeout=1) <Response [200]>
If you set the time to a very small number, for example
requests.get('http://github.com', timeout=0.001)
If there is no connection within the timeout time, a Timeout exception will be thrown.
9. Use session in access
Initialize a session object first,
s = requests.Session()
Then use this session object for access, r = s. post (url, data = user)
Access Renren's network to obtain the most recent access in the home page, and then access to view more visits to read more recent visits.
More visits are to access the http://www.renren.com/myfoot.do with session
# Coding: utf-8import requestsimport reurl = r 'HTTP: // www.renren.com/ajaxLogin'user = {'e-mail ': 'e-mail', 'Password': 'pass'} s = requests. session () r = s. post (url, data = user) html = r. textvisit = [] first = re. compile (R' </span> <span class = "time-tip first-tip"> <span class = "tip-content"> (. *?) </Span> ') second = re. compile (R' </span> <span class = "time-tip"> <span class = "tip-content"> (. *?) </Span> ') third = re. compile (R' </span> <span class = "time-tip last-second-tip"> <span class = "tip-content"> (. *?) </Span> ') last = re. compile (R' </span> <span class = "time-tip last-tip"> <span class = "tip-content"> (. *?) </Span> ') visit. extend (first. findall (html) visit. extend (second. findall (html) visit. extend (third. findall (html) visit. extend (last. findall (html) for I in visit: print iprint below are more recent visits to 'vm = s. get ('HTTP: // www.renren.com/myfoot.do') fm = re. compile (R' "name ":"(. *?) "') Visitmore = fm. findall (vm. text) for I in visitmore: print I
10. requests-cookies
Cookies store and save the values of each item like a dictionary, such as our username, password, and login information. when a Web page is loaded again, you can find related information from cookies and avoid re-entering the value assignment process.
When using get or other requests in requests, we can also grant cookie information. For example, we can obtain the cookies requested when a webpage is loaded from a browser and grant requests the same permission to use them again.
Cookies can be transmitted by adding the cookie = {key: value} parameter to the requests request.
import requestsurl = 'http://httpbin.org/cookies'cookies = dict(cookies_are='working')r = requests.get(url, cookies=cookies)r.text#'{"cookies": {"cookies_are": "working"}}'
It is very easy to query the cookies of a request, just like using the cookies property like getting headers:
url = 'http://example.com/some/cookie/setting/url'r = requests.get(url)r.cookies['example_cookie_name']# 'example_cookie_value'
The following function can break down the cookie string obtained by the browser into a dictionary to help us simulate requests.
def browsercookiesdict(s): '''Covert cookies string from browser to a dict''' ss=s.split(';') outdict={} for item in ss: i1=item.split('=',1)[0].strip() i2=item.split('=',1)[1].strip() outdict[i1]=i2 return outdict