python socket.error: [Errno 10054] The remote host forced the shutdown of an existing connection. Problem Solutions:
Use Python to read Web pages a few days ago. Because of a large number of use of a site urlopen operation, so it will be identified as an attack by the site. Downloads are not allowed at times. After causing Urlopen (), Request.read () has been stuck there. Errno 10054 will be thrown at the end.
This error is connection reset by peer. That is, the legendary remote host resets this connection. The reason may be that the socket timeout is too long, or it may be after request = Urllib.request.urlopen (URL), there is no request.close () operation, or there may be no sleep for a few seconds, causing the site to identify this behavior as an attack.
Specific solutions such as the following code:
ImportSocketImportTime Timeout= 20socket.setdefaulttimeout (Timeout)#This sets the time-out for the entire socket layer. In subsequent files, if you use the socket again, you do not have to setSleep_download_time = 10time.sleep (sleep_download_time)#time to set yourself up here .Request = Urllib.request.urlopen (URL)#here is the URL to read the contentContent = Request.read ()#read, usually reported in here abnormalRequest.close ()#Remember to closeImportSocketImportTimetimeout= 20socket.setdefaulttimeout (Timeout)#This sets the time-out for the entire socket layer. In subsequent files, if you use the socket again, you do not have to setSleep_download_time = 10time.sleep (sleep_download_time)#time to set yourself up here .Request = Urllib.request.urlopen (URL)#here is the URL to read the contentContent = Request.read ()#read, usually reported in here abnormalRequest.close ()#Remember to close
Because the read () operation after Urlopen is actually called some function of the socket layer. So set the socket default timeout time, you can let the network itself off. You do not have to wait at read ().
Of course you can also write a few try,except on the outer layer, for example:
Try: Time.sleep (self.sleep_download_time) Request=urllib.request.urlopen (URL) content=Request.read () request.close ( )exceptUnicodedecodeerror as E:Print('-----Unicodedecodeerrorurl:', URL)exceptUrllib.error.URLError as E:Print("-----Urlerrorurl:", URL)exceptSocket.timeout as E:Print("-----Socket Timout:", URL)Try: Time.sleep (self.sleep_download_time) Request=urllib.request.urlopen (URL) content=Request.read () request.close ( )exceptUnicodedecodeerror as E:Print('-----Unicodedecodeerrorurl:', URL)exceptUrllib.error.URLError as E:Print("-----Urlerrorurl:", URL)exceptSocket.timeout as E:Print("-----Socket Timout:", URL)
Generally speaking, there is no problem. I tested the download of thousands of pages before I spoke. However, if the download is thousands of, I did the next test, Ms will still jump out of this exception. It may be that the time of Time.sleep () is too short or the network is suddenly interrupted. I tested it with Urllib.request.retrieve () and found that there was always a case of failure when I downloaded the data constantly.
The simple way to do this is: the first reference to my article: Python checkpoint simple implementation. Make a checkpoint first. Then the code above will run out of the exception while true. See Pseudo-code below:
defDownload_auto (downloadlist,fun,sleep_time=15): whileTrue:Try:#outsourcing One-layer tryValue = Fun (downloadlist,sleep_time)#Here's the fun is your download function, I when the function pointer is passed in. #only normal execution can exit. ifValue = =util.success: Break except:#If it happens, 10054 or IOError or xxxerror.Sleep_time + = 5#sleep more than 5 seconds and re-execute the above download. Because of the checkpoint, the above program will continue to execute from the point where the exception was thrown. Prevents program outages caused by network connection instability. Print('Enlarge sleep Time:', Sleep_time)defDownload_auto (downloadlist,fun,sleep_time=15): whileTrue:Try:#outsourcing One-layer tryValue = Fun (downloadlist,sleep_time)#Here's the fun is your download function, I when the function pointer is passed in. #only normal execution can exit. ifValue = =util.success: Break except:#If it happens, 10054 or IOError or xxxerror.Sleep_time + = 5#sleep more than 5 seconds and re-execute the above download. Because of the checkpoint, the above program will continue to execute from the point where the exception was thrown. Prevents program outages caused by network connection instability. Print('Enlarge sleep Time:', Sleep_time)
However, for the corresponding page can not be found, but also to do another treatment:
#Print Download InformationdefReporthook (Blocks_read, Block_size, total_size):if notBlocks_read:Print('Connection opened') ifTotal_size <0:Print('Read%d blocks'%blocks_read)Else: #if not found, the page does not exist, may TotalSize is 0, can not calculate the percentage Print('downloading:%d MB, totalsize:%d MB'% (blocks_read*block_size/1048576.0,total_size/1048576.0)) defDownload (path,url):#url = ' Http://downloads.sourceforge.net/sourceforge/alliancep2p/Alliance-v1.0.6.jar ' #filename = url.rsplit ("/") [-1] Try: #python's own download functionurllib.request.urlretrieve (URL, path, Reporthook)exceptIOError as E:#if not found, it seems to cause ioerror. Print("Download"Url"\nerror:", E)Print("done:%s\ncopy to:%s"%(Url,path))#Print Download InformationdefReporthook (Blocks_read, Block_size, total_size):if notBlocks_read:Print('Connection opened') ifTotal_size <0:Print('Read%d blocks'%blocks_read)Else: #if not found, the page does not exist, may TotalSize is 0, can not calculate the percentage Print('downloading:%d MB, totalsize:%d MB'% (blocks_read*block_size/1048576.0,total_size/1048576.0)) defDownload (path,url):#url = ' Http://downloads.sourceforge.net/sourceforge/alliancep2p/Alliance-v1.0.6.jar ' #filename = url.rsplit ("/") [-1] Try: #python's own download functionurllib.request.urlretrieve (URL, path, Reporthook)exceptIOError as E:#if not found, it seems to cause ioerror. Print("Download"Url"\nerror:", E)Print("done:%s\ncopy to:%s"% (Url,path))
If you're still having problems ... Please comment on other solutions on the note.
Original website: http://www.cnblogs.com/mmix2009/archive/2013/07/30/3224954.html
Python socket timeout setting errno10054