Production and consumption patterns of Python parallel tasks

I. Producer / consumer model

Concept: The producer generates a piece of data and puts it in buffer , while the consumer pulls and consumes the data from the buffer

Understanding : Like in the life of manufacturers to produce products, customers buy consumption of these products, buffer is the storage of goods warehouse.

Two. implementation of the producer/consumer model in Python

related modules : Queue module

Brief Introduction : In Python, queues are one of the most common forms of exchanging data between threads. The queue module is a module in Python that provides queued operations.

principle : It creates a "queue" object (that is, buffer to hold the data), and then continuously generates the data and stores it in the queue, while also continuously

To remove data from the queue.

Specific functions :

(1) Create a Queue object

1 Import Queue 2 >>> q = queue.queue ()

Note: The queue length can be unlimited or limited. The queue length can be set through the optional parameter maxsize of the queue's constructor. If MaxSize is less than 1, the queue length is infinite, for example:

(2) Depositing data into the queue

Method: q.put (item, Block=true, Timeout=none)

>>> q.put (' a ')

Note: put () has two parameters, the first item is required, the value of the inserted item, the second block is an optional parameter, and the default is 1.

If the queue is currently empty and the Block is the 1,put () method, the calling thread pauses until a data cell is vacated. If the block is the 0,put method, the full exception is thrown.

(3) Extracting data from the queue

Method: q.get (Block=true, Timeout=none)

>>> Q.get ()

Note: The Get method optional parameter is block, which defaults to true.

If the queue is empty and the Block is True,get (), the calling thread is paused until a project is available. If the queue is empty and the block is false, the queue throws an empty exception.

Common methods in Queue.queue:

Q.qsize () returns the size of the queue Q.empty () returns True if the queue is empty, and vice versa Falseq.full ()  returns True if the queue is full, and vice versa.　 　　Corresponds to maxsize size q.get ([block[, timeout]]) Get queue, timeout wait time q.get_nowait ()    quite q.get (False) non-blocking q.put (item,timeout) Write queue, timeout wait time q.put_nowait (item)    quite Q.put (item, False) Q.task_done ()  After completing a work, the function sends a signal to the queue that the task has completed Q.join ( )  indicates that wait until the queue is empty before performing another operation

Example test:

#!/usr/bin/env python#Coding=utf-8Importthreading, timeImportQueue#Import Message Queuing moduleImportRandom#The random number module is imported to simulate the situation where the producer and consumer speed are inconsistent. q = queue.queue ()         #instantiate a queue object that can be used when there are multiple threads sharing one thingdefProducer ():#producer Functions     forIinchRange (20): q.put (i) #put the results in Message Queuing        Print '[+] Product%s'%I Time.sleep (Random.randrange (3))#production speed of the producer, within 3sdefConsumer ():#Consumer FunctionsCount =0 whileCount < 20: Data = Q.get ()    #take the results stored in the message queue        Print '[-] consume%s'%Data Count+ = 1Time.sleep (Random.randrange (4))#Consumer's consumption rate, within 4sProducter= Threading. Thread (target =Producer) Consumer= Threading. Thread (target =Consumer) Producter.start () Consumer.start ()

Operation Result:

　　　　　　　

Production and consumption patterns of Python parallel tasks