Python3 Crawler Requests Module Usage Summary

Requests is a third-party module that needs to be imported if it is to be used. Requests can also be said to be an upgraded version of the Urllib module, more convenient to use.

This is an example of using Urllib.

Import urllib.request Import  'http://www.weather.com.cn/data/sk/101190408.html'= Urllib.request.urlopen (URL)# send request result = Res.read (). Decode ()# Get results, The result is a byte type that requires decode ()print(json.loads (Result))

The following is the use of the requests module.

Supported requests:

Requests.get (' Https://github.com/timeline.json ') #GET请求
Requests.post ("Http:xxx.xx.com/post") #POST请求
Requests.put ("Http:xxx.xx.com/put") #PUT请求
Requests.delete ("Http:xxx.xx.com/delete") #DELETE请求
Requests.head ("Http:xxx.xx.com/head") #HEAD请求
Requests.options ("Http:xxx.xx.com/get") #OPTIONS请求

To send a GET request:

ImportRequests,jsonurl='Http://api.xx.cn/api/user/stu_info?stu_name=hi'req= Requests.get (URL)#send a GET requestPrint(Req.text)#the JSON string is returned directly to get the result.Print(Type (Req.text))#StrPrint(Json.loads (Req.text))#JSON to DictionaryPrint(Req.json ())#Get the result is a dictionary, only the JSON string is returned to use Req.json ()Print(Type (Req.json ()))#Dict

Send a POST request

' Http://api.xxx.cn/api/user/login '  = {'username':'aa','passwd  ':'123'= Requests.post (url,data)#  Send a POST request, the first parameter is the URL, the second parameter is the requested data print(Req.json ())

Send data formatted as JSON

URL ='Http://api.xxx.cn/api/user/add_stu'Data= {    "name":"AA",    "Grade":"BB",    "Phone": 13512530000,}req= Requests.post (Url,json=data)#To send a POST request, the first parameter is the URL, the second parameter is the requested data, and the JSON that is sent is written json=dataPrint(Req.json ())

Send a request with a cookie

URL ='Http://api.xx.cn/api/user/gold_add'Data= {'stu_id': 231,'Gold':' -'}cookie= {'AA':'3d867b361afdaac1381b02ae746c7278'}#key is the login username, value is signreq = Requests.post (Url,data,cookies=cookie)#Add a cookiePrint(Req.json ())

Send a request with header

' Http://api.xxx.cn/api/user/all_stu '  = {'user-agent':'Chrome'= Requests.get (Url,headers = header) Print (Req.json ())

Uploading files

' Http://api.xxx.cn/api/file/file_upload '  = open (R'D:\aa.jpg','rb')#  Picture to specify to open R =requests.post (url,files={'file': F})in binary modeprint (R.json ())

Download files, images, videos

URL ='Https://images2017.cnblogs.com/blog/412654/201712/412654-20171213115213238-464712233.png'R=requests.get (URL)Print(R.status_code)#GET request Status CodePrint(r.content)#gets the returned result in binary format.FW = Open ('bt.jpg','WB')#Current Path#FW = open (R ' d:\bt.jpg ', ' WB ') #指定绝对路径Fw.write (r.content)#writing binary format content to a fileFw.close ()

Crawler, save the page to local

' http://www.cnblogs.com/nancyzhu/p/8029994.html '  == open ('page.html','wb') f.write (r.content) f.close ()

Python3 Crawler Requests Module Usage Summary