Python implements multi-threaded HTTP Downloader

Source: Internet
Author: User
Tags ranges python script

This article describes writing a multithreaded HTTP downloader using Python and generating an. exe executable file.

Environment : Windows/linux + python2.7.x

Single Thread

introduce a single thread before you introduce multithreading. The idea of writing a single thread is:

    1. Parse URL;
    2. Connect to the Web server;
    3. Constructs an HTTP request packet;
    4. Download the file.

This is followed by a code description.

parsing URLs

Parsed by the user input URL. If the resolved path is empty, the assignment is '/'; If the port number is empty, the value is "80"; the file name of the downloaded file can be changed according to the user's wishes (input ' y ' means change, input other means no change required).

Several analytic functions are listed below:

#parsing host and pathdefAnalyhostandpath (totalurl): Protocol,s1=Urllib.splittype (totalurl) host, Path=urllib.splithost (S1)ifPath = ="': Path='/'    returnHost, Path#Parse PortdefAnalysisport (Host): Host, Port=Urllib.splitport (host)ifPort isNone:return80returnPort#Parse filenamedefanalysisfilename (path): filename= Path.split ('/') [-1]    if '.'  not inchFileName:returnNonereturnFileName 

Connecting to a Web server

Using the socket module, the host and port are connected to the Web server based on the parse URL, the code is as follows:

Import Socket  from Import  == socket.socket (socket.af_inet,socket. Sock_stream) s.connect ((IP, port))print"success Connected webserver!! "
constructing an HTTP request package

based on the parse URL, the path, host, Port constructs an HTTP request packet.  

 from Import   "'\r\n\r\n'   
Download File

according to the constructed HTTP request packet, send the file to the server and fetch the "content-length" of the response message header.  

def GetLength (self):        s.send (packet)        print"send success! "                   = s.recv (1024x768) Print buf = Re.compile (R '          content-length: (\d*) ' )        = int (p.findall (BUF) [0])        return length, buf

Download the file and calculate the time it took to download it.  

defDownload (self): file= Open (Self.filename,'WB') Length,buf=self.getlength () Packetindex= Buf.index ('\r\n\r\n') BUF= Buf[packetindex+4:] File.write (buf) sum=Len (BUF) while1: buf= S.RECV (1024) file.write (buf) sum= Sum +Len (BUF)ifSum >=Length: Break        Print "success!!"if __name__=="__main__": Start=time.time () down=Downloader () () End=time.time ()Print "The time spent is%f s"% (End-start)

Operation Result:


Grab the "content-length" field in the header of the response packet, combined with the number of threads, and lock the segment download. Unlike a single thread, this is where all the code is consolidated into one file, with more Python-brought modules in the code.

get "content-length":  

def GetLength (self):         = Urllib2.build_opener ()        = (self.url)        = ()        = Int ( Meta.getheaders ("content-length") [0])        return
  According to the resulting length, the number of threads is divided into the range:     
 def   Get_range (self): ranges  = [] length  = Self.getlength () offset  = Int (i NT (length)/ Self.threadnum)  for  i in   range (self.threadnum):  if  i = = (Self.threadnum-1 *offset, "  else: Ranges.append ((i  *offset, (i+1) *offset )  return  ranges 

Implement multi-threaded download, when writing content to the file, loads locks to the line, and use with lock instead of Lock.acquire () ... lock.release (); use () to set the file offset address to ensure the accuracy of the write file.

defDownloadthread (self,start,end): Req=Urllib2. Request (self.url) req.headers['Range'] ='bytes=%s-%s'%(start, end) F=Urllib2.urlopen (req) offset=Start Buffer= 1024 while1: (buffer)if  notBLOCK: BreakWith (offset) self.file.write (block) offs ET= offset +Len (block)defDownload (self): filename=self.getfilename () self.file= open (filename,'WB') Thread_list=[] n= 1 forRaninchSelf.get_range (): Start, End=ranPrint 'starting:%d Thread'%N N+ = 1Thread= Threading. Thread (target=self.downloadthread,args=(Start,end)) Thread.Start () thread_list.append (thread) forIinchThread_list:i.join ()Print 'Download%s success!'%(Self.file) self.file.close ()
Operation Result:

CONVERT (*.py) file to (*.exe) executable file

When you've written a tool, how do you get people who don't have Python installed to use this tool? This requires converting the. py file to an. exe file.

The Python py2exe module is used here for the first time, so it is introduced:

Py2exe is a tool that translates a Python script into an executable file (*.exe) that can be executed independently on Windows, so that you can run it on Windows without having to install Python. The corresponding version can be downloaded in.

Next, under the same directory as the, create the file and write:

 from Import Setup Import py2exesetup (console=[""])  

Then execute the command:Python Py2exe

Generate the Dist folder where the MultiTjhreadDownload.exe file is located and click to run:

Above is my use of Python implementation of multi-threaded HTTP Downloader process, you have any good ideas, but also look at the liberal enlighten!

Links to the project: Https://

My github:

Python implements multi-threaded HTTP Downloader

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.