Python programming-Explanation of how to add cookies to network requests, pythonrequests

Python programming-Explanation of how to add cookies to network requests, pythonrequests

Ah, I haven't learned crawler for a long time. Now I want to pick it up again. I found that the crawler I learned was a little rough. I didn't even know how to add cookies to requests. Not too much nonsense.

We usually use requests to get the network content, which is easy to get with several lines of code, for example:

import requests res=requests.get("https://cloud.flyme.cn/browser/index.jsp") print res.content

There are only three lines of code. However, the simplicity is simple, and there are still many problems.

First, the request here is only a get method and does not solve the cookie problem.

Then, the garbled problem is not solved here. (Garbled characters are abnormal points in python) In fact, it is not difficult to add cookies in the requests get method:

res=requests.get("https://cloud.flyme.cn/browser/index.jsp",cookies=cookies) 

The problem we face is how to construct cookies here.

Cookies are data in dictionary format.

When we browse the webpage, we can view the cookies on the accessed webpage in the review elements, mostly like this:

sn_openNetBySms=%23810EBMA3TE53; sn_map=810EBMA3TE53; DSESSIONID=f1987887-3d1d-4a85-ad75-c6270e588290; JSESSIONID=; _uid=; _keyLogin=; _rmtk=; _uticket=ns_0393027c2f9f686e3499e8ebb8d1d622; _ckk=ns_397a592791064029bf1336eff1cf516e; ucuid=8a135520affa423584307f6e2c210f02; _domain=cloud.flyme.cn; _islogin=true; lang=zh_CN; JSESSIONID=1gmfzynp0ns6s1u6a92xkqgi6q 

Oh, is that what people see? No. This is the encrypted data. Similarly, we can pass the encrypted data to the cookies parameter of the get method. The server will automatically decrypt the encrypted data and check whether the data is correct.

As for how to view the cookies on the accessed webpage, you can use the browser's review function, F12 or right-click Review (Chrome) to open it. View the Headers content of the link in the network. You can also use the software to capture packets, such as Fiddler web Debugger.

Hosts file.
Then,

F=open(r'test.txt ', 'R') # Open the stored cookies file cookies ={}# initialize the cookie dictionary variable for line in f. read (). split (';'): # split and read by character: # if it is set to 1, the string will be split into two parts: name, value = line. strip (). split ('=', 1) cookies [name] = value # add content to dictionary cookies

In this case, we add cookies to the get method:

res=requests.get("https://cloud.flyme.cn/browser/index.jsp",cookies=cookies) 

In this case, the obtained res. content is the content obtained by accessing the webpage after we add cookies to get.

Summary

The above is all the details about how to add cookies to the network request of requests in python programming. I hope it will be helpful to you. If you are interested, you can continue to refer to this site: compile function parsing of python Regular Expression re, Python_LDA implementation method explanation, Python exploration to modify Python search path, etc, you are welcome to leave a message for discussion. Thank you for your support!