Python crawler Development (i)-------requests module

Requests:

Requests function (You can view detailed functions through Pycharm):-requests.get () Requests.post () Requests.put () Requests.delet () ... requests . Request ()

Parameters: Requests.request ()

-method: How to submit

-url: Submit Address

-params parameters passed in the URL, get

Cases:

1 Requests.request (2 3METHOD ="Get",4URL ="www.ekankan.com"5params = {"K1":"v1","K2":"v2"}6 )7 #Http://www.www.ekankan.com?k1=v1&k2=v2

-data: Data passed in the request body

1 Requests.request (2  3METHOD ="Get",4URL ="www.ekankan.com",5params = {"K1":"v1","K2":"v2",6data = {'uers':'Alex','pwd':'123'}   7 }8  9 Ten----------------------->#equivalent to converting a dictionary into a string One Requests.request ( A   -METHOD ="Get", -URL ="www.ekankan.com", theparams = {"K1":"v1","K2":"v2", -data ='uers=alex&pwd=123' -}

-json data passed in the request body

1 Requests.request (2METHOD ='Get',3URL ='www.ekankan.com',4params = {'K1':'v1','K2':'v2'},5JSON = {'uers':'Alex','pwd':'123'}6     7     #Internal conversions: "{' uers ': ' Alex ', ' pwd ': ' 123 '}"8 9)

#思考 How to set up the anti-crawl mechanism: (the source of the Referer record your last landing site)

-headers Request Header

1 Requests.request (2METHOD ='Get',3URL ='www.ekankan.com',4params = {'K1':'v1','K2':'v2'},5JSON = {'uers':'Alex','pwd':'123'}6 7headers = {8       'Referer':'www.ekankan.com/'9        'user-agent':'-------'#property that indicates what browser you are using to accessTen } One      #Internal conversions: Forge the last access record A   -)

-cookies Request Server

Put it in the headers, and send it over.

The above is the most important requests function!!!!

More parameters

Session get Cookie and get GPSD grant permission

1 ImportRequests2 3Session =requests. Session ()4 5 ## # # #首先登陆任何页面, get Cookies6i1 = session.get (url ="https://dig.chouti.com/link/vote?linksId=21518036")7 ## # # #用户登录, carry the last cookie, and authorize the GPSD in the cookie in the background8I2 =Session.post (9URL ="Http://dig.chouti.com/login",TenData= { One         'Phone':'XXX', A         'Password':'Jshfkaj', -         'Onemonth':'1' -  the     } -  -  - ) +i3 =Session.post ( -URL ='https://dig.chouti.com/link/vote?linksId=21518036' + ) A Print(I3.text)

Python crawler Development (i)-------requests module