Python socket timeout setting errno 10054

Python socket. error: [Errno 10054] the remote host forces an existing connection to be closed. Solution:

I used python to read web pages a few days ago. Because a website uses a large number of urlopen operations, it will be identified as an attack by that website. Sometimes download is no longer allowed. As a result, request. read () remains stuck there after urlopen. Errno 10054 will be thrown.

This error is caused by connection reset by peer. That is, the remote host has reset the connection. The reason may be that the socket timeout is too long, or request = urllib. request. after urlopen (url), no request is made. close () operation, or it may take a few seconds for the website to identify this behavior as an attack.

The specific solution is as follows:

01. import socket 02. import time 03. timeout = 20 04. socket. setdefatimetimeout (timeout) # Set the timeout time for the entire socket layer. If socket is used in subsequent files, you do not have to set it again. sleep_download_time = 10 06. time. sleep (sleep_download_time) # set the time here to 07. request = urllib. request. urlopen (url) # Here is the url 08 of the content to be read. content = request. read () # read, usually exception 09. request. close () # Remember to close

Because the read () operation after urlopen actually calls some functions at the socket layer. Therefore, you can disable the network by setting the default socket timeout. You do not have to wait at read.

Of course, you can write a few more try and try t on the outer layer, for example:

try:   time.sleep(self.sleep_download_time)   request = urllib.request.urlopen(url)   content = request.read()   request.close()    except UnicodeDecodeError as e:        print('-----UnicodeDecodeError url:',url)    except urllib.error.URLError as e:   print("-----urlError url:",url)  except socket.timeout as e:   print("-----socket timout:",url) 

Generally, there is no problem. I tested the download of thousands of webpages and then said this. However, if you download tens of thousands of files, I did a test and ms will still jump out of this exception. It may be that the time. sleep () is too short, or the network is suddenly interrupted. I tested urllib. request. retrieve () and found that data downloading continuously always fails.

A simple solution is as follows: first, refer to my article: python checkpoint simple implementation. Make a Check Point first. Then, run the exception Section Code while True. See the following pseudocode:

Def Download_auto (downloadlist, fun, sleep_time = 15): while True: try: # outsourcing level try value = fun (downloadlist, sleep_time) # fun here is your download function, when the function pointer is passed in. # Exit only after normal execution. If value = Util. SUCCESS: break failed T: # If 10054 or IOError or XXXError sleep_time + = 5 # Sleep for 5 seconds, run the download command again. because of the checkpoint, the above program will continue to execute from where an exception is thrown. This prevents program interruptions caused by unstable network connections. Print ('enlarge sleep time: ', sleep_time)

However, if you cannot find the corresponding webpage, you need to do another thing:

# Print download information def reporthook (blocks_read, block_size, total_size): if not blocks_read: print ('Connection opened') if total_size <0: print ('read % d blocks '% blocks_read) else: # If the page cannot be found and does not exist, the totalsize may be 0. The percentage print ('downloading: % d MB, totalsize: % d MB '% (blocks_read * block_size/1048576.0, total_size/1048576.0) def Download (path, url): # url = 'HTTP: // downloads.sourceforge.net/sourceforge/alliancepcep /Alliance-v1.0.6.jar '# filename = url. rsplit ("/") [-1] try: # download function urllib provided by python. request. urlretrieve (url, path, reporthook) failed t IOError as e: # if it cannot be found, it may cause an IOError. Print ("download", url, "/nerror:", e) print ("Done: % s/nCopy to: % s" % (url, path ))

If you still have problems... please comment on other solutions.