7.2.6-Concurrent multithreaded producer, consumer

Tag: Python balanced signal res LAN multi Erro concurrent programming

Introduction of a producer consumer model

Why use a producer consumer model

Producers refer to the task of producing data, the consumer is the task of processing data, in concurrent programming, if the producer processing speed quickly, and the consumer processing speed is very slow, then the producer must wait for the consumer to finish processing, can continue production data. Similarly, consumers must wait for producers if their processing power is greater than that of producers. To solve this problem, the producer and consumer models were introduced.

What is the producer and consumer model

The producer-consumer model solves the problem of strong coupling between producers and consumers through a container. Producers and consumers do not communicate with each other directly, and through the blocking queue to communicate, so producers do not have to wait for consumer processing after the production of data, directly to the blocking queue, consumers do not find producers to data, but directly from the blocking queue, the blocking queue is equivalent to a buffer, Balance the processing power of producers and consumers.

This blocking queue is used to decouple producers and consumers.

Realization of two-producer consumer model

Practice a producer consumer model based on the queue of the previous bar learning

 fromMultiprocessingImportProcess,queueImport Timedefproduct (q): forIinchRange (10): Time.sleep (0.5)        Print("production of%s ge Bun"%i) q.put (i)defConsumer (q): whileTrue:time.sleep (1) Res=Q.get ()ifRes isNone:Print("The buns are finished .")             Break        Print("Consumption--%s ge Bun"%Res)if __name__=="__main__": Q=Queue () P1= Process (target=product,args=(q,)) C1= Process (target=consumer,args=(q,)) P1.start () C1.start () P1.join () Q.put (None) # produce everything in the producer finally add a None at the end of the queue, to not block,

# The problem at this point is that the main process will never end, because producer P is finished after production, but consumer c after taking the empty Q, it has been in the dead loop and stuck in Q.get () this step. The solution is to let the producer after the production, the queue to send an end signal, so that the consumer after receiving the end signal can break out of the dead loop

The result is

production of 0 ge Buns produced1ge buns have consumed-0 GE Buns were produced2GE Bun has produced3ge buns have consumed-1GE Bun has produced4GE Bun has produced5ge buns have consumed-2GE Bun has produced6GE Bun has produced7ge buns have consumed-3GE Bun has produced8GE Bun has produced9ge buns have consumed-4ge buns have consumed-5ge buns have consumed-6ge buns have consumed-7ge buns have consumed-8ge buns have consumed-9GE steamed buns are finished .

View Code

Multiple producers, consumers

 fromMultiprocessingImportProcess,queueImport Timedefproduct (q): forIinchRange (10): Time.sleep (0.5)        Print("production of%s buns"%i) q.put (i)defConsumer (q): whileTrue:time.sleep (1) Res=Q.get ()ifRes isNone:Print("The buns are finished .")             Break        Print("consumption of--%s buns"%Res)if __name__=="__main__": Q=Queue () P1= Process (target=product,args=(q,)) P2= Process (target=product,args=(q,)) P3= Process (target=product,args=(q,)) C1= Process (target=consumer,args=(q,)) C2= Process (target=consumer,args=(q,)) P1.start () P2.start () P3.start () C1.start () C2.start () P1.join () P2.join () P3.join () Q.put (None) Q.put (None) #最后是2个None, a few consumers on a few of None

Results

C:\Python35\python.exe d:/luffy/Fourth module/10 producer consumer model. PY produced a total of 0 buns produced 0 buns produced 0 buns produced1a bun was produced1a bun was produced1a bun for consumption-0 Buns were consumed-0 Buns were produced2a bun was produced2a bun was produced2a bun was produced3a bun was produced3a bun was produced3a bun for consumption-0 Buns were consumed-1a bun was produced4a bun was produced4a bun was produced4a bun was produced5a bun was produced5a bun was produced5a bun for consumption-1a bun for consumption-1a bun was produced6a bun was produced6a bun was produced6a bun was produced7a bun was produced7a bun was produced7a bun for consumption-2a bun for consumption-2a bun was produced8a bun was produced8a bun was produced8a bun was produced9a bun was produced9a bun was produced9a bun for consumption-2a bun for consumption-3a bun for consumption-3a bun for consumption-3a bun for consumption-4a bun for consumption-4a bun for consumption-4a bun for consumption-5a bun for consumption-5a bun for consumption-5a bun for consumption-6a bun for consumption-6a bun for consumption-6a bun for consumption-7a bun for consumption-7a bun for consumption-7a bun for consumption-8a bun for consumption-8a bun for consumption-8a bun for consumption-9a bun for consumption-9a bun for consumption-9Steamed Buns are all eaten.

Run ResultsBut the above solution, when there are multiple producers and multiple consumers, we need a very low way to solve, a few consumers will need to send several end signals: quite low, for example

Joinablequeue ([maxsize])

这就像是一个Queue对象，但队列允许项目的使用者通知生成者项目已经被成功处理。通知进程是使用共享的信号和条件变量来实现的。

Parameter introduction

maxsize是队列中允许最大项数，省略则无大小限制。

Method Introduction

JoinableQueue的实例p除了与Queue对象相同的方法之外还具有：q.task_done()：使用者使用此方法发出信号，表示q.get()的返回项目已经被处理。如果调用此方法的次数大于从队列中删除项目的数量，将引发ValueError异常q.join():生产者调用此方法进行阻塞，直到队列中所有的项目均被处理。阻塞将持续到队列中的每个项目均调用q.task_done（）方法为止

Realization of producer consumer model based on Joinablequeue

 fromMultiprocessingImportProcess,joinablequeueImport Timedefproduct (q): forIinchRange (2): Time.sleep (0.5)        Print("production of%s buns"%i) q.put (i) q.join ()defConsumer (q): whileTrue:res=Q.get ()ifRes isNone:Print("The buns are finished .")             BreakTime.sleep (1)        Print("consumption of--%s buns"%res) Q.task_done ()if __name__=="__main__": Q=joinablequeue () P1= Process (target=product,args=(q,)) P2= Process (target=product,args=(q,)) P3= Process (target=product,args=(q,)) C1= Process (target=consumer,args=(q,)) C2= Process (target=consumer,args=(q,)) C1.daemon=True C2.daemon=True P1.start () P2.start () P3.start () C1.start () C2.start () P1.join () P2.join () P3.join () 

The running result is

111 Buns consumed-- 0 buns consumed -- 0 Buns consumed-- 0 buns consumed --1< c7> a bun consumption --1 buns consumed --1 buns

Run Results

Summary of producer Consumer models

1. There are two types of roles in the program

一类负责生产数据（生产者）一类负责处理数据（消费者）

2, the introduction of producer consumer model in order to solve the problem is

平衡生产者与消费者之间的速度差程序解开耦合

3. How to realize the producer consumer model

生产者<--->队列<--->消费者

7.2.6-Concurrent multithreaded producer, consumer