python-multi-process Message Queuing

Message Queuing

Message Queuing is the container that holds messages during the transmission of messages

The most classic usage of Message Queuing is that consumers and producers pass messages through the message pipeline, and consumers and generators are different processes. Producers write messages to pipelines, consumers read messages from pipelines

The operating system provides a number of mechanisms for inter-process communication, and the multiprocessing module provides two ways of queue and pipe to achieve

First, use the queue inside the multiprocessing to implement Message Queuing

Q = Queue

Q.put (data)

data = Q.get ()

Example:

from multiprocessing import queue, processdef  write (q):     for i in ["A", "B", "C", "D"]:         q.put (i)         print ("Put {0} to  queue ". Format (i)) Def read (q):    while 1:         result = q.get ()         print ("get {0 } from queue ". Format (Result)) Def main ():     q = queue ()      pw = process (target=write,args= (q,))     pr = process (target=read,args= (q,))     pw.start ()     pr.start ()      pw.join ()     pr.terminate () if __name__ ==  "__main__":     main () 

Operation Result:

Put A to queue

Put B to queueget a from queue

Get B from queue

Put C to queue

Put D to queue

Get C from queue

Get D from queue

Second, through the pipe inside the multiprocessing to achieve Message Queuing

1) Pipe method return (CONN1,CONN2) represents the two end of a pipe. The pipe method has the duplex parameter, if the Duplux parameter is true (the default), then the pipeline is full duplex, that is, both CONN1 and CONN2 can send and receive. Duplex is responsible for false,conn1 receiving the message, CONN2 is responsible for issuing the message

2) The Send and Recv methods are methods for sending and receiving messages, respectively. The Close method means that the pipeline is closed and the pipe is closed when the message is received.

Example:

From multiprocessing import process,pipeimport timedef proc1 (pipe):     for i in xrange (1,10):         pipe.send (i)          time.sleep (3)         print (" Send {0} to pipe ". Format (i)) def proc2 (pipe):    n = 9     while n>0:        result =  PIPE.RECV ()         time.sleep (3)          print ("Recv {0} from pipe". Format (Result))          n -= 1if __name__ ==  "__main__":    pipe =  Pipe (Duplex=false)     print (Type (pipe))     p1 = process ( Target=proc1,args= (Pipe[1],)) &Nbsp;   p2 = process (target=proc2,args= (pipe[0),)     p1.start ()     p2.start ()     p1.join ()     p2.join ()      pipe[0].close ()     pipe[1].close ()

Operation Result:

<type ' tuple ' >

Send 1 to Pipe

Recv 1 from Pipe

Send 2 to Pipe

Recv 2 from Pipe

Recv 3 from Pipe

Send 3 to Pipe

Send 4 to PIPERECV 4 from pipe

Send 5 to Pipe

Recv 5 from Pipe

Recv 6 from Pipe

Send 6 to Pipe

Send 7 to Pipe

Recv 7 from Pipe

Send 8 to Pipe

Recv 8 from Pipe

Recv 9 from Pipesend 9 to pipe

Third, queue module

Python provides a queue module to specifically implement Message Queuing:

The queue object implements a FIFO queue (as well as the LIFO, priority queue). Queue only Gsize a constructor, which specifies the capacity of the queue, when specified as 0 represents unlimited capacity. As long as the following member functions are available:

Queue.gsize (): Returns the current space of the message queue. The returned value is not necessarily reliable.

Queue.empty (): Determines whether the message queue is empty and returns True or false. is also unreliable

Queue.full (): Determines whether the message is full

Queue.put (Item,block=true,timeout=none): Stores data in message queues. Blocks can control whether the block is blocked, and timeout controls the wait time when blocking. If it is not blocked or timed out, it will cause a full exception.

Queue.put_nowait (item): Equivalent to put (Item,false)

Queue.get (Block=true,timeout=none): Gets a message, other equivalent put

Use two functions to determine whether a message corresponds to the completion of a task:

Queue.task_done (): The thread that receives the message indicates that the task that corresponds to the message has been completed by calling this letter

Queue.join (): Actually means waiting for the queue to be empty before performing another operation

python-multi-process Message Queuing