I am very confused about a curl issue. I used curl to simulate Baidu's access Promotion. my goal is to use cookies to obtain the json data in it, as shown below: 1. log on to the Baidu promotion website through simulated login. 2. Then, use the cookie and post parameters to obtain json data. 3. Parse and import the obtained json data into the database. The problem is: when I got half of the data, the code showed that I could not get the json data. Then I re-execute the script to find a question about curl that makes me very difficult.
I used curl to simulate the promotion of Baidu. my purpose is to obtain the json data in the cookie. the current situation is as follows:
1. log onto the Baidu promotion website through simulated login
2. then obtain json data through the cookie and post parameters.
3. Parse and import the received json data to the database
Now the problem is: when I get half of the data into the database, the code shows that the json data cannot be obtained. Then I re-execute the script and the result is that the first half of the data cannot be obtained. (although I have already stored this half of the data, I can re-obtain it, in principle, json data is returned ).
My understanding is that the curl access may be faulty when the json data is retrieved (but this should not be a code issue because half of the data has been obtained before, this is like a sudden disconnection from the Internet)
I would like to ask you if this problem has not occurred when you use curl. please answer!
------ Solution --------------------
All data is obtained, data integrity is judged, and then the write operation is performed ..
------ Solution --------------------
This is not the case. it is obtained only when the sent request completely returns the curl, and the post request receives a response, either completely or completely. This is not like displaying web pages. sometimes only some of them are loaded and some of them are not displayed.
------ Solution --------------------
Yes, your request is sent too fast, and the other party blocks you temporarily,
Sleep between requests several seconds .....
