Python implementation multithreaded HTTP Downloader sample

This article describes writing a multithreaded HTTP downloader using Python and generating an. exe executable file.

Environment: Windows/linux + python2.7.x

Single Thread

Introduce a single thread before you introduce multithreading. The idea of writing a single thread is:

1. Parse the URL;

2. Connect to the Web server;

3. Construct the HTTP request packet;

4. Download the file.

This is followed by a code description.

Parsing URLs

Parsed by the user input URL. If the resolved path is empty, the assignment is '/'; If the port number is empty, the value is "80"; the file name of the downloaded file can be changed according to the user's wishes (input ' y ' means change, input other means no change required).

Several analytic functions are listed below:

#解析host和pathdef Analyhostandpath (totalurl):  protocol,s1 = Urllib.splittype (totalurl)  host, Path = Urllib.splithost (S1)  if Path = = ":    path = '/'  return host, path# parse Portdef analysisport (host):  host, Port = Urllib.splitport (host)  if Port is None:    return to return  port# parse filenamedef analysisfilename (path):  filename = path.split ('/') [-1]  if '. ' Not in filename:    return None  return filename

Connecting to a Web server

Using the socket module, the host and port are connected to the Web server based on the parse URL, the code is as follows:

Import socketfrom analysisurl Import Port,hostip = Socket.gethostbyname (host) s = Socket.socket (socket.af_inet,socket. Sock_stream) s.connect ((IP, port)) print "Success connected webserver!! "

Constructing an HTTP request package

Based on the parse URL, the path, host, Port constructs an HTTP request packet.

From Analysisurl import path, host, Portpacket = ' GET ' + path + ' Http/1.1\r\nhost: ' + host + ' \r\n\r\n '

Download file

According to the constructed HTTP request packet, send the file to the server and fetch the "content-length" of the response message header.

def getlength (self):    s.send (packet)    print "Send Success!"    BUF = S.recv (1024x768)    print buf    p = re.compile (R ' content-length: (\d*) ')    Length = Int (P.findall (BUF) [0])    return length, buf

Download the file and calculate the time it took to download it.

def download (self):    file = open (Self.filename, ' WB ')    length,buf = Self.getlength ()    Packetindex = Buf.index (' \r\n\r\n ')    BUF = buf[packetindex+4:]    file.write (buf)    sum = Len (buf) while    1:      buf = S.recv (1024x768)      File.write ( BUF)      sum = sum + len (buf)      if sum >= length:        break    print "success!!" if __name__ = = "__main__":  start = Time.time () Down  = Downloader ()  down.download ()  end = Time.time ( )  print "The time spent on the"%f S "% (End-start)

Multithreading

Grab the "content-length" field in the header of the response packet, combined with the number of threads, and lock the segment download. Unlike a single thread, this is where all the code is consolidated into one file, with more Python-brought modules in the code.

Get "content-length":

def getlength (self):    opener = Urllib2.build_opener ()    req = Opener.open (self.url)    meta = Req.info ()    length = Int (meta.getheaders ("Content-length") [0])    return length

According to the resulting length, the number of threads is divided into the range:

def get_range (self):    ranges = []    length = self.getlength ()    offset = int (int (length)/self.threadnum)    For I in Range (self.threadnum):      If i = = (self.threadnum-1):        ranges.append ((I*offset, '))      else:        Ranges.append ((I*offset, (i+1) *offset))    return ranges

Implement multi-threaded download, when writing content to the file, loads locks to the line, and use with lock instead of lock.acquire () ... lock.release (); use File.seek () to set the file offset address to ensure the accuracy of the write file.

def downloadthread (self,start,end):    req = urllib2. Request (self.url)    req.headers[' Range ' = ' bytes=%s-%s '% (start, end)    f = urllib2.urlopen (req)    offset = Start    buffer = 1024x768    1:      block = f.read (buffer)      if not block: Break with      Lock:        Self.file.seek (offset)        self.file.write (block)        offset = offset + len (block)  def download (self):    filename = self.getfilename ()    self.file = open (filename, ' WB ')    thread_list = []    n = 1 for ran in self    . Get_range ():      start, end = ran      print ' starting:%d thread '% n      n + = 1      thread = Threading. Thread (target=self.downloadthread,args= (start,end))      Thread.Start ()      thread_list.append (thread)    For i in Thread_list:      i.join ()    print ' Download%s success! '% ( Self.file)    Self.file.close ()

Operation Result:

Convert (*.py) file to (*.exe) executable file

When you've written a tool, how do you get people who don't have Python installed to use this tool? This requires converting the. py file to an. exe file.

The Python py2exe module is used here for the first time, so it is introduced:

Py2exe is a tool that translates a Python script into an executable file (*.exe) that can be executed independently on Windows, so that you can run it on Windows without having to install Python.

Next, under the same directory as the multithreaddownload.py, create the mysetup.py file and write:

From Distutils.core import setupimport py2exesetup (console=["multithreaddownload.py"])

Then execute the command: Python mysetup.py py2exe

Generate the Dist folder where the MultiTjhreadDownload.exe file is located and click to run:

Demo Download Address: Httpfiledownload_jb51.rar

The above is the whole content of this article, I hope that everyone's learning has helped, but also hope that we support topic.alibabacloud.com.