Python requests request stuck problem

Recently often received feedback from a crawler tool program is not working properly, need to download information data is not included in time.

At first thought it might be a problem with the machine, and occasionally the machine would get stuck in the middle of the program running.

But as the situation became more and more frequent, I had to troubleshoot the problem.

By viewing the log information that the program runs, the discovery program is always stuck in the requests request step.

It makes me wonder why this step will get stuck, I have set the time-out, supposedly timed out the program will enter the exception ah, but what the exception information is not output.

Find out the relevant information, and found that this situation is true.

Requests is written on the document, timeout is not a time limit on the entire response download; Rather, an exception is raised if the server have not issued a response for timeout seconds (more precisely, if no bytes ha ve been received on the underlying socket for timeout seconds). If No timeout is specified explicitly, requests does not time out.

The approximate meaning is that the timeout set on the requests request is not the time limit for the entire response download, and the timeout here should be understood as the time to receive the data after the connection with the server and timeout if the specified time is exceeded. This is very different from what I have understood before.

So how do we solve this problem?

There are two ways to fix this, one is to change DNS, and recommend Ali's DNS because they think the card here is stuck because of the DNS resolution there.

The other is to set the global socket time-out, by

Import socket
Socket.setdefaulttimeout (Time)

These two lines of code to solve.

By testing, the program will still get stuck after using the second scenario. (Is it really stuck in DNS parsing?) If so, can you only change DNS? Is there any other way? ）

Because the network connection is not very well understood, it is unclear why. This question should be left here first and then supplemented when it is resolved.

Python requests request stuck problem