Python Thread Programming-synchronous queue

Python Thread Programming-synchronous queue

We often use two threads in the producer/consumer relationship to process data in a shared buffer. For example, a producer thread accepts user data into a shared buffer and waits for a consumer thread to extract and process the data. However, if the buffer is too small and the producer and consumer asynchronous threads are not at the same speed, one thread may wait for the other. To minimize the waiting time of threads that share resources and work at the same speed, we can use a "queue" to provide additional buffers.

Create a "queue" Object

import Queuemyqueue = Queue.Queue(maxsize = 10)

The Queue. Queue class is the synchronization implementation of a Queue. The queue length can be unlimited or limited. You can set the Queue length through the optional parameter maxsize of the Queue constructor. If maxsize is smaller than 1, the queue length is infinite.

Put a value into the queue

myqueue.put(10)

Call the put () method of the queue object to insert a project at the end of the team. Put () has two parameters. The first item is required and is the value of the inserted project. The second block is an optional parameter. The default value is 1. If the queue is empty and the block is 1, The put () method suspends the calling thread until a data unit is empty. If the block is 0, the put method will cause a Full exception.

Extract A value from the queue

myqueue.get()

Call the get () method of the queue object to delete the queue header and return a project. The optional parameter is block. The default value is 1. If the queue is empty and the block is 1, get () will suspend the calling thread until a project is available. If the block is 0, the queue will cause an Empty exception.

Let's use an example to demonstrate how to use Queue

#!/usr/bin/env pythonimport Queueimport threadingimport urllib2import time          hosts = ["http://yahoo.com", "http://google.com.hk", "http://amazon.com",          "http://ibm.com", "http://apple.com"]          queue = Queue.Queue()          class ThreadUrl(threading.Thread):      """Threaded Url Grab"""       def __init__(self, queue):              threading.Thread.__init__(self)              self.queue = queue                 def run(self):           while True:                #grabs host from queue                host = self.queue.get()                     #grabs urls of hosts and prints first 1024 bytes of page                url = urllib2.urlopen(host)                print url.read(1024)                        #signals to queue job is done                self.queue.task_done()    start = time.time()def main():     #spawn a pool of threads, and pass them queue instance      for i in range(5):            t = ThreadUrl(queue)            t.setDaemon(True)            t.start()                    #populate queue with data         for host in hosts:            queue.put(host)       #wait on the queue until everything has been processed            queue.join()          main()print "Elapsed Time: %s" % (time.time() - start)

This mode is common and recommended when using threads in Python. The procedure is described as follows:

1. Create a Queue. Queue () instance and fill it with data.
2. Pass the instance with filled data to the Thread class, which is created by inheriting threading. Thread.
3. Generate a daemon thread pool.
4. Each time a project is retrieved from the queue, the data in the thread and the run method are used to execute the corresponding work.
5. After this task is completed, use the queue. task_done () function to send a signal to the completed queue of the task.
6. The join Operation on the queue actually means that the main program is exited when the queue is empty.

When using this mode, note that by setting the daemon thread to true, the main thread or program can exit only when the daemon thread is active. This method creates a simple way to control the program flow, because before exiting, you can perform the join operation on the queue or wait until the queue is empty.

Join ()

Keep blocked until all projects in the queue are processed. When a project is added to the queue, the total number of unfinished tasks increases. When the user thread calls task_done () to retrieve the project and complete all the work, the total number of unfinished tasks is reduced. When the total number of unfinished tasks is reduced to zero, join () ends the blocking state.