5 Concurrent Programming--Queue & producer Consumer Models

1, the introduction of the queue

Processes are isolated from each other, and to implement interprocess communication (IPC), the Multiprocessing module supports two forms: queues and pipelines, both of which use message passing

Create a queue class (the underlying is implemented as a pipe and lock):

Queue([maxsize]):创建共享的进程队列，Queue是多进程安全的队列，可以使用Queue实现多进程之间的数据传递。

Parameter description:

MaxSize is the maximum number of items allowed in a queue, and no size limit is omitted. But it needs to    be clear:1, the queue is stored in the message    rather than big Data 2, the queue is occupied by memory space, so maxsize even if no size limit is limited by the memory size

Main methods:

The Q.put method is used to insert data into the queue. The Q.get method can read from the queue and delete an element.

 fromMultiprocessingImportProcess,queueq=queue (3)#put, get, Put_nowait,get_nowait,full,emptyQ.put (1) Q.put (2) Q.put (3)Print(Q.full ())#full of#Q.put (4) #再放就阻塞住了Print(Q.get ())Print(Q.get ())Print(Q.get ())Print(Q.empty ())#it's empty.#print (Q.get ()) #再取就阻塞住了True123True

II. producer Consumer Model 1, producer consumer Model introduction why use producer consumer models

Producers refer to the task of producing data, the consumer is the task of processing data, in concurrent programming,

If producers are processing quickly and consumers are slow to handle, then producers must wait for consumers to finish the process,

To continue to produce data. Similarly, consumers must wait for producers if their processing power is greater than that of producers.

To solve this problem, the producer and consumer models were introduced.

What is the producer and consumer model

The producer-consumer model solves the problem of strong coupling between producers and consumers through a container. Producers and consumers do not communicate with each other directly, and through the blocking queue to communicate, so producers do not have to wait for consumer processing after the production of data, directly to the blocking queue, consumers do not find producers to data, but directly from the blocking queue, the blocking queue is equivalent to a buffer, Balance the processing power of producers and consumers.

This blocking queue is used to decouple producers and consumers.

2, the producer consumer model realizes 2.1 introduces the model (produces one consumes one)

Import Timedefproducer (): forIinchRange (3): Res= f"Bun {i}"Time.sleep (0.5)        Print(f"the producer produced the {res}") Consumer (RES)defConsumer (RES): Time.sleep (1)    Print(f"the consumer ate {res}")if __name__=='__main__': Producer () producer produced buns 0 consumers ate buns 0 producers produced buns1The consumer eats the steamed bun1The producer produced the Buns2The consumer eats the steamed bun2

View Code2.2 Implementation of the producer consumer model, but there are small problems the main process will never end the consumer does not know that the producer has finished, has been in a waiting state, the dead loop

 fromMultiprocessingImportProcess,queueImport Timedefproducer (q): forIinchRange (3): Res= f"Bun {i}"Time.sleep (0.5)        Print(f"the producer produced the {res}")        #Save the production to the queueq.put (RES)defConsumer (q): whileTrue:#consumers have been receivingres =Q.get () time.sleep (1)        Print(f"the consumer ate {res}")if __name__=='__main__': Q=Queue () P1= Process (target=producer,args=(q,)) P2= Process (target=consumer,args=(q,)) P1.start () P2.start ( )Print('Master'The main producer produced the Bun 0 producer produced the Bun1The producer produced the Buns2consumers ate buns 0 consumers ate buns1The consumer eats the steamed bun2

View Code

The problem at this point is that the main process will never end, because producer P is finished after production, but consumer C has been in the dead loop and stuck in the Q.get () step after the Q has been emptied.

The solution is to let the producer after the production, the queue to send an end signal, so that the consumer after receiving the end signal can break out of the dead loop

2.3, the solution-in fact, our idea is to send the end signal only

Queue FIFO

 fromMultiprocessingImportProcess,queueImport Timedefproducer (q): forIinchRange (3): Res= f"Bun {i}"Time.sleep (0.5)        Print(f"the producer produced the {res}")        #Save the production to the queueq.put (RES)defConsumer (q): whileTrue:#consumers have been receivingres =Q.get ()ifres = =None: BreakTime.sleep (1)        Print(f"the consumer ate {res}")if __name__=='__main__': Q=Queue () P1= Process (target=producer,args=(q,)) P2= Process (target=consumer,args=(q,)) P1.start () P2.start () P1.join ()#The main process waits for the P1 child process to finish executing-that is, the producer is finishedq.put (None)Print('Master'The producer produced the Bun 0 producers produced the bun1The producer produced the Buns2consumers ate buns 0 main consumers ate buns1The consumer eats the steamed bun2

View Code

But the above solution, when there are multiple producers and multiple consumers, we need a very low way to solve, a few consumers will need to send several end signals: quite low, for example

if __name__=='__main__': Q=Queue ()#producers: the chefsP1=process (target=producer,args= (q,'Egon1','steamed Bun')) P2=process (target=producer,args= (q,'Egon2','Bones')) P3=process (target=producer,args= (q,'Egon3','swill'))    #consumers: Those who are foodiesC1=process (target=consumer,args= (q,'alex1')) C2=process (target=consumer,args= (q,'Alex2'))    #StartP1.start () P2.start () P3.start () C1.start () C2.start () P1.join () P2.join () P3.join () q.put (none) Q.put (None) Q.put (none)Print('Master')

2.4 Joinablequeue ([maxsize])

5 Concurrent Programming--Queue & producer Consumer Models