Python implements multi-threaded download instances based on queue and threading, and queuethreading

Python implements multi-threaded download instances based on queue and threading, and queuethreading

The example in this article describes how to implement multi-threaded download using python Based on queue and threading. The specific method is as follows:

The main code is as follows:

  #download worker   queue_download = Queue.Queue(0)   DOWNLOAD_WORKERS = 20   for i in range(DOWNLOAD_WORKERS):     DownloadWorker(queue_download).start() #start a download worker        for md5 in MD5S:     queue_download.put(md5)   for i in range(DOWNLOAD_WORKERS):     queue_download.put(None) 

Downloadworkers. py
Class inherits threading. Thread, and the run method is overloaded. threading. Thread. _ init _ (self) is called in _ init ),
Implement time-consuming operations in the run Method

Import threading import Queue import md5query import DOM import OS, sys class DownloadWorker (threading. thread): "def _ init _ (self, queue):" "Constructor" "self. _ queue = queue threading. thread. _ init _ (self) def run (self): while 1: md5 = self. _ queue. get () if md5 is None: break # reached end of queue # this is a time-cost produce self. _ down (md5) print "task:", md5, "finished" def _ down (self, md5): config = {'input': sys. stdin, 'output ':'. /samples ', 'location': 'xxx', 'has-fn': False, 'options': {'connect. timeout': 60, 'timeout': 3600}, 'log': file('logs.txt ', 'w'),} print 'Download % s... '% (md5) try: data = downloadproc (config ['location'], config ['options']) # My download process if data: dom, fileData = md5query. splited (data) filename = md5 if config ['has-fn ']: filename =' % s _ % s' % (md5, dom. nodeValue2 ('xxxxxxx ',''). encode ('utf-8') # This is my download method f = file (OS. path. join (config ['output'], filename), 'w') f. write (fileData) f. close () print '% s \ Tok' % (md5) else: print> config ['log'],' % s \ t % s' % (md5, 'failed') failed t Exception, e: print> config ['log'], '% s \ t % s' % (md5, str (e ))

I hope this article will help you with Python programming.

In the python threading module, after multiple threads are generated, how does one obtain the string returned after the thread is executed?

Multi-threaded/multi-process communication or callback, rather than directly returning results. This is easy to understand, because if you use the returned results to assign values to a variable, you must wait until the function ends and your program will be blocked, this eliminates the significance of multi-threaded/multi-process blocking.
Communication can be event-driven or transmitted using thread-safe data structures (such as Queue, 0mq, and rabbitMQ ), callback means that after a program is executed, you can call another function to process the next step.

A multithreaded Website access tool written in the threading module of Python, but when I set a large number of threads

Check whether your interaction is wrong. If different threads access the same part, this issue may easily occur.