Python socket timeout setting errno 10054

Python socket.error: [Errno 10054] The remote host forced the shutdown of an existing connection. Problem Solutions:

Use Python to read Web pages a few days ago. Because of a large number of use of a site urlopen operation, so it will be identified as an attack by the site. Downloads are not allowed at times. After causing Urlopen (), Request.read () has been stuck there. Errno 10054 will be thrown at the end.

This error is connection reset by peer. That is, the legendary remote host resets this connection. The reason may be that the socket timeout is too long, or it may be after request = Urllib.request.urlopen (URL), there is no request.close () operation, or there may be no sleep for a few seconds, causing the site to identify this behavior as an attack.

Specific solutions such as the following code:

01.import socket 02.import Time 03.timeout =  04.socket.setdefaulttimeout (timeout) #这里对整个socket层设置超时时间. If you use the socket again in subsequent files, you do not have to set  05.sleep_download_time = 06.time.sleep (sleep_download_time) #这里时间自己设定  07. Request = Urllib.request.urlopen (URL) #这里是要读取内容的url  08.content = Request.read () #读取, which is usually reported as an exception here  

Because the read () operation after Urlopen is actually called some function of the socket layer. So set the socket default timeout time, you can let the network itself off. You do not have to wait at read ().

Of course you can also write a few try,except on the outer layer, for example:

Try:   time.sleep (self.sleep_download_time)   request = Urllib.request.urlopen (URL)   content = Request.read ()   request.close ()    except Unicodedecodeerror as E:        print ('-----unicodedecodeerror url: ', url)    Except Urllib.error.URLError as E:   print ("-----urlerror URL:", url)  except Socket.timeout as E:   

Generally speaking, there is no problem. I tested the download of thousands of pages before I spoke. However, if the download is thousands of, I did the next test, Ms will still jump out of this exception. It may be that the time of Time.sleep () is too short or the network is suddenly interrupted. I tested it with Urllib.request.retrieve () and found that there was always a case of failure when I downloaded the data constantly.

The simple way to do this is: the first reference to my article: Python checkpoint simple implementation. Make a checkpoint first. Then the code above will run out of the exception while true. See Pseudo-code below:

def download_auto (downloadlist,fun,sleep_time=15): While   True:         Try: # Outsource a layer        of try value = Fun (Downloadlist, Sleep_time) # Here's the fun is your download function, I when the function pointer passed in.       # only normal execution can exit.        if value = = util.success: Break     except: # if 10054 or IOError or xxxerror       sleep_time + = 5 #多睡5秒 occurred, Re-execute the above download. Because of the checkpoint, the above program will continue to execute from the point where the exception was thrown. Prevents program outages caused by network connection instability.       

However, for the corresponding page can not be found, but also to do another treatment:

# Print Download Info  def reporthook (Blocks_read, Block_size, total_size):   if not blocks_read:     print (' Connection Opened ')   if total_size < 0:     print (' Read%d blocks '% blocks_read)   else:     # If not found, the page does not exist, Maybe totalsize is 0, can't calculate percent      print (' downloading:%d MB, totalsize:%d mb '% (Blocks_read*block_size/1048576.0,total_ size/1048576.0))      def Download (path,url): #url = ' http://downloads.sourceforge.net/sourceforge/alliancep2p/ Alliance-v1.0.6.jar '   #filename = Url.rsplit ("/") [-1]    try:   # Python's own download function      Urllib.request.urlretrieve (URL, path, reporthook)   except IOError as e: # If not found, it seems to raise IOError.     print ("Download", URL, "/nerror:", E)   

If you're still having problems ... Please comment on other solutions on the note.

