Python 3 Concurrent Programming Multi-process queue (recommended)

Python 3 Concurrent Programming Multi-process queue (recommended)

Processes are isolated from each other, and to implement interprocess communication (IPC), the Multiprocessing module supports two forms: queues and pipelines, both of which are delivered using messaging.

Can put any type of data into the queue

Create a queue class (the underlying is implemented as a pipe and lock):

1 queue ([MaxSize]): Creates a shared process queue, which is a multi-process secure queue that enables data transfer between multiple processes using the queue.

Parameter description:

1 MaxSize is the maximum number of items allowed in a queue, and no size limit is omitted.

Method Description:

1, the Main method:

1 The Q.put method is used to insert data into the queue, and the put method has two optional parameters: Blocked and timeout. If blocked is true (the default) and timeout is a positive value, the method blocks the time specified by timeout until the queue has the remaining space. If timed out, a Queue.full exception is thrown. If blocked is false, but the queue is full, an Queue.full exception is thrown immediately. 2 the Q.get method can read from the queue and delete an element. Similarly, the Get method has two optional parameters: Blocked and timeout. If blocked is true (the default) and timeout is a positive value, then no element is taken within the wait time, and a Queue.empty exception is thrown. If blocked is false, there are two cases where the queue has a value that is available, and the value is immediately returned, otherwise the Queue.empty exception is thrown immediately if it is empty.3 q.get_nowait (): Same as Q.get (False)4 q.put_nowait (): Same as Q.put (False)5 Q.empty (): When calling this method, q is null to return true, and the result is unreliable, such as in the process of returning true, if the item is added to the queue. 6 Q.full (): When this method is called, Q is full to return true, and the result is unreliable, for example, in the process of returning true, if the items in the queue are taken away. 7Q.qsize (): Returns the correct number of current items in the queue, and the results are unreliable, for the same reason as Q.empty () and Q.full ()

2. Other methods (understand):

1 q.cancel_join_thread (): The background thread is not automatically connected when the process exits. You can prevent the Join_thread () method from blocking 2 q.close (): Close the queue and prevent more data from being added to the queue. Calling this method, the background thread will continue to write to the data that has been queued but not yet written, but will close as soon as this method completes. If q is garbage collected, this method is called. Closing a queue does not produce any type of data end signal or exception in the queue consumer. For example, if a consumer is being blocked on a get () operation, shutting down a queue in a producer does not cause the get () method to return an error. 3 Q.join_thread (): The background thread that connects the queue. This method is used to wait for all queue items to be consumed after the Q.close () method is called. By default, this method is called by all processes that are not the original creator of Q. Calling the Q.cancel_join_thread method can prohibit this behavior

3. Application:

" "the multiprocessing module supports two main forms of interprocess communication: Pipelines and queues are based on message delivery, but queue interface 1: You can put any type of data into the queue 2 queues: FIFO" " fromMultiprocessingImportProcess,queueImporttimeq=queue (3)#put, get, Put_nowait,get_nowait,full,emptyQ.put (3) Q.put (3) Q.put (3)Print(Q.full ())#full ofPrint(Q.get ())Print(Q.get ())Print(Q.get ())Print(Q.empty ())#it's empty.

View Code

4. Producer Consumer Model

Using producer and consumer patterns in concurrent programming can solve most concurrency problems. This mode improves the overall processing speed of the program by balancing the productivity of the production line and the consuming thread.

5. Why use producer and consumer models

In the world of threads, the producer is the thread of production data, and the consumer is the thread of consumption data. In multithreaded development, producers have to wait for the consumer to continue producing data if the producer is processing fast and the consumer processing is slow. Similarly, consumers must wait for producers if their processing power is greater than that of producers. To solve this problem, the producer and consumer models were introduced.

6. What is the producer consumer model

The producer-consumer model solves the problem of strong coupling between producers and consumers through a container. Producers and consumers do not communicate with each other directly, and through the blocking queue to communicate, so producers do not have to wait for consumer processing after the production of data, directly to the blocking queue, consumers do not find producers to data, but directly from the blocking queue, the blocking queue is equivalent to a buffer, Balance the processing power of producers and consumers.

7, based on the queue to implement the producer consumer model

 fromMultiprocessingImportProcess,queueImportTime,random,osdefConsumer (q): whileTrue:res=Q.get () time.sleep (Random.randint (1,3))        Print('\033[45m%s Eat%s\033[0m'%(Os.getpid (), res))defproducer (q): forIinchRange (10): Time.sleep (Random.randint (1,3)) Res='Bun%s'%I q.put (res)Print('\033[44m%s produced%s\033[0m.'%(Os.getpid (), res))if __name__=='__main__': Q=Queue ()#producers: the chefsP1=process (target=producer,args=(q,))#consumers: Those who are foodiesC1=process (target=consumer,args=(q,))#StartP1.start () C1.start ( )Print('Master')

Producer Consumer Model

The problem at this point is that the main process will never end, because producer P is finished after production, but consumer C has been in the dead loop and stuck in the Q.get () step after the Q has been emptied.

The solution is to let the producer after the production, the queue to send an end signal, so that the consumer after receiving the end signal can break out of the dead loop.

 fromMultiprocessingImportProcess,queueImportTime,random,osdefConsumer (q): whileTrue:res=Q.get ()ifRes isNone: Break #the end signal is received .Time.sleep (Random.randint (1,3))        Print('\033[45m%s Eat%s\033[0m'%(Os.getpid (), res))defproducer (q): forIinchRange (10): Time.sleep (Random.randint (1,3)) Res='Bun%s'%I q.put (res)Print('\033[44m%s produced%s\033[0m.'%(Os.getpid (), res)) Q.put (None)#Send End Signalif __name__=='__main__': Q=Queue ()#producers: the chefsP1=process (target=producer,args=(q,))#consumers: Those who are foodiesC1=process (target=consumer,args=(q,))#StartP1.start () C1.start ( )Print('Master')

The producer sends the end signal after the production is complete none

Note: The end signal is none, not necessarily by the producer, the main process can also be sent, but the main process needs to wait for the producer to send the signal after the end.

 fromMultiprocessingImportProcess,queueImportTime,random,osdefConsumer (q): whileTrue:res=Q.get ()ifRes isNone: Break #the end signal is received .Time.sleep (Random.randint (1,3))        Print('\033[45m%s Eat%s\033[0m'%(Os.getpid (), res))defproducer (name,q): forIinchRange (2): Time.sleep (Random.randint (1,3)) Res='%s%s'%(name,i) q.put (res)Print('\033[44m%s produced%s\033[0m.'%(Os.getpid (), res))if __name__=='__main__': Q=Queue ()#producers: the chefsP1=process (target=producer,args= ('steamed Bun', Q)) P2=process (target=producer,args= ('Bones', Q)) P3=process (target=producer,args= ('swill', Q)) #consumers: Those who are foodiesC1=process (target=consumer,args=(q,)) C2=process (target=consumer,args=(q,))#StartP1.start () P2.start () P3.start () C1.start () P1.join ()#must ensure that all production is completed before the end signal should be sentP2.join () p3.join () q.put (None)#There are several producers that should send a few end signals to noneQ.put (None)#Send End SignalQ.put (None)#Send End Signal    Print('Master')

The main process sends the end signal to none after the producer has finished producing

In fact, our idea is to send the end signal only, there is another kind of queue provides this mechanism

# Joinablequeue ([maxsize]): This is like a queue object, but the queue allows the consumer of the project to notify the creator that the project has been successfully processed. The notification process is implemented using shared signals and condition variables.  # parameter description:    maxsize is the maximum number of items allowed in a queue, and no size limit is omitted.     # Method Description:     Joinablequeue Instance p has the same method as the queue object:    Q.task_done (): The consumer uses this method to signal that the returned item of Q.get () has been processed. If the number of times this method is called is greater than the number of items removed from the queue, the ValueError exception is thrown    Q.join (): The producer calls this method to block until all items in the queue are processed. Blocking will persist until each item in the queue calls the Q.task_done () method

 fromMultiprocessingImportProcess,joinablequeueImportTime,random,osdefConsumer (q): whileTrue:res=Q.get () time.sleep (Random.randint (1,3))        Print('\033[45m%s Eat%s\033[0m'%(Os.getpid (), res)) Q.task_done ()#send a signal to Q.join () to prove that a data has been taken awaydefproducer (name,q): forIinchRange (10): Time.sleep (Random.randint (1,3)) Res='%s%s'%(name,i) q.put (res)Print('\033[44m%s produced%s\033[0m.'%(Os.getpid (), res)) Q.join ()if __name__=='__main__': Q=Joinablequeue ()#producers: the chefsP1=process (target=producer,args= ('steamed Bun', Q)) P2=process (target=producer,args= ('Bones', Q)) P3=process (target=producer,args= ('swill', Q)) #consumers: Those who are foodiesC1=process (target=consumer,args=(q,)) C2=process (target=consumer,args=(q,)) C1.daemon=True C2.daemon=True#StartP_l=[P1,P2,P3,C1,C2] forPinchP_l:p.start () p1.join () P2.join () P3.join ( )Print('Master')         #main process, such as--->p1,p2,p3---->c1,c2    #P1,p2,p3 is over, prove C1,c2 's all done. P1,p2,p3 sent to the queue of data    #thus C1,C2 also has no value, and should end with the end of the main process, so set up as a daemon

View Code

Python 3 Concurrent Programming Multi-process queue (recommended)