Python implements multi-threaded HTTP Downloader

This article describes writing a multithreaded HTTP downloader using Python and generating an. exe executable file.

Environment : Windows/linux + python2.7.x

Single Thread

introduce a single thread before you introduce multithreading. The idea of writing a single thread is:

1. Parse URL;
2. Connect to the Web server;
3. Constructs an HTTP request packet;
4. Download the file.

This is followed by a code description.

parsing URLs

Parsed by the user input URL. If the resolved path is empty, the assignment is '/'; If the port number is empty, the value is "80"; the file name of the downloaded file can be changed according to the user's wishes (input ' y ' means change, input other means no change required).

Several analytic functions are listed below:

#parsing host and pathdefAnalyhostandpath (totalurl): Protocol,s1=Urllib.splittype (totalurl) host, Path=urllib.splithost (S1)ifPath = ="': Path='/'    returnHost, Path#Parse PortdefAnalysisport (Host): Host, Port=Urllib.splitport (host)ifPort isNone:return80returnPort#Parse filenamedefanalysisfilename (path): filename= Path.split ('/') [-1]    if '.'  not inchFileName:returnNonereturnFileName 

Connecting to a Web server

Using the socket module, the host and port are connected to the Web server based on the parse URL, the code is as follows:

Import Socket  from Import  == socket.socket (socket.af_inet,socket. Sock_stream) s.connect ((IP, port))print"success Connected webserver!! "

constructing an HTTP request package

based on the parse URL, the path, host, Port constructs an HTTP request packet.  

 from Import   "'\r\n\r\n'  　

Download File

according to the constructed HTTP request packet, send the file to the server and fetch the "content-length" of the response message header.  

def GetLength (self):        s.send (packet)        print"send success! "                   = s.recv (1024x768) Print buf = Re.compile (R '          content-length: (\d*) ' )        = int (p.findall (BUF) [0])        return length, buf

Download the file and calculate the time it took to download it.  

defDownload (self): file= Open (Self.filename,'WB') Length,buf=self.getlength () Packetindex= Buf.index ('\r\n\r\n') BUF= Buf[packetindex+4:] File.write (buf) sum=Len (BUF) while1: buf= S.RECV (1024) file.write (buf) sum= Sum +Len (BUF)ifSum >=Length: Break        Print "success!!"if __name__=="__main__": Start=time.time () down=Downloader () down.download () End=time.time ()Print "The time spent is%f s"% (End-start)

Operation Result:

Multithreading

Grab the "content-length" field in the header of the response packet, combined with the number of threads, and lock the segment download. Unlike a single thread, this is where all the code is consolidated into one file, with more Python-brought modules in the code.

get "content-length":  

def GetLength (self):         = Urllib2.build_opener ()        = opener.open (self.url)        = req.info ()        = Int ( Meta.getheaders ("content-length") [0])        return

  According to the resulting length, the number of threads is divided into the range:     

 def   Get_range (self): ranges  = [] length  = Self.getlength () offset  = Int (i NT (length)/ Self.threadnum)  for  i in   range (self.threadnum):  if  i = = (Self.threadnum-1 *offset, "  else: Ranges.append ((i  *offset, (i+1) *offset )  return  ranges 

Implement multi-threaded download, when writing content to the file, loads locks to the line, and use with lock instead of Lock.acquire () ... lock.release (); use File.seek () to set the file offset address to ensure the accuracy of the write file.

defDownloadthread (self,start,end): Req=Urllib2. Request (self.url) req.headers['Range'] ='bytes=%s-%s'%(start, end) F=Urllib2.urlopen (req) offset=Start Buffer= 1024 while1: Block=f.read (buffer)if  notBLOCK: BreakWith Lock:self.file.seek (offset) self.file.write (block) offs ET= offset +Len (block)defDownload (self): filename=self.getfilename () self.file= open (filename,'WB') Thread_list=[] n= 1 forRaninchSelf.get_range (): Start, End=ranPrint 'starting:%d Thread'%N N+ = 1Thread= Threading. Thread (target=self.downloadthread,args=(Start,end)) Thread.Start () thread_list.append (thread) forIinchThread_list:i.join ()Print 'Download%s success!'%(Self.file) self.file.close ()

Operation Result:

CONVERT (*.py) file to (*.exe) executable file

When you've written a tool, how do you get people who don't have Python installed to use this tool? This requires converting the. py file to an. exe file.

The Python py2exe module is used here for the first time, so it is introduced:

Py2exe is a tool that translates a Python script into an executable file (*.exe) that can be executed independently on Windows, so that you can run it on Windows without having to install Python. The corresponding version can be downloaded in.

Next, under the same directory as the multithreaddownload.py, create the mysetup.py file and write:

 from Import Setup Import py2exesetup (console=["multithreaddownload.py"])  

Then execute the command:Python mysetup.py Py2exe

Generate the Dist folder where the MultiTjhreadDownload.exe file is located and click to run:

Above is my use of Python implementation of multi-threaded HTTP Downloader process, you have any good ideas, but also look at the liberal enlighten!

Links to the project: Https://github.com/Yabea/HttpFileDownload

My github:https://github.com/yabea.

Python implements multi-threaded HTTP Downloader