Python implements multi-thread download for instances based on queue and threading

This article mainly introduces how to download an instance using multiple threads in python based on queue and threading. it is a practical technique, for more information about how to download multiple threads in python based on queue and threading, see the following example. The specific method is as follows:

The main code is as follows:

  #download worker   queue_download = Queue.Queue(0)   DOWNLOAD_WORKERS = 20   for i in range(DOWNLOAD_WORKERS):     DownloadWorker(queue_download).start() #start a download worker        for md5 in MD5S:     queue_download.put(md5)   for i in range(DOWNLOAD_WORKERS):     queue_download.put(None) 

Downloadworkers. py
Class inherits threading. Thread, and the run method is overloaded. threading. Thread. _ init _ (self) is called in _ init ),
Implement time-consuming operations in the run method

Import threading import Queue import md5query import DOM import OS, sys class DownloadWorker (threading. thread): "def _ init _ (self, queue):" "Constructor" "self. _ queue = queue threading. thread. _ init _ (self) def run (self): while 1: md5 = self. _ queue. get () if md5 is None: break # reached end of queue # this is a time-cost produce self. _ down (md5) print "task:", md5, "finished" def _ down (self, md5): config = {'input': sys. stdin, 'output ':'. /samples ', 'Location': 'XXX', 'has-fn': False, 'options': {'connect. timeout': 60, 'timeout': 3600}, 'log': file('logs.txt ', 'w'),} print 'download % s... '% (md5) try: data = downloadproc (config ['location'], config ['options']) # My download process if data: dom, fileData = md5query. splited (data) filename = md5 if config ['has-fn ']: filename =' % s _ % s' % (md5, dom. nodeValue2 ('xxxxxxx ',''). encode ('utf-8') # This is my download method f = file (OS. path. join (config ['output'], filename), 'w') f. write (fileData) f. close () print '% s \ Tok' % (md5) else: print> config ['log'],' % s \ t % s' % (md5, 'failed') failed t Exception, e: print> config ['log'], '% s \ t % s' % (md5, str (e ))

I hope this article will help you with Python programming.