Introduction to pipe and queue of multiprocessing multi-process communication in python

In the past two days, the python multiprocessing multi-process module has been warmed up. The pipe and queue ipc methods are displayed. What is ipc? Ipc is the communication mode between processes. The commonly used half is socke, rpc, pipe, and message queue.


Now we will discuss pipe and queue.

The code is as follows:

Copy code

# Coding: UTF-8 Import multiprocessing Import time   Def proc1 (pipe ): While True: For I in xrange (10000 ): Print "send % s" % I Pipe. send (I) Time. sleep (1)   Def proc2 (pipe ): While True: Print 'proc2 received: ', pipe. recv () Time. sleep (1)   Def proc3 (pipe ): While True: Print 'proc3 received: ', pipe. recv () Time. sleep (1) # Build a pipe Pipe = multiprocessing. Pipe () Print pipe   # Pass an end of the pipe to process 1 P1 = multiprocessing. Process (target = proc1, args = (pipe [0],) # Pass the other end of the pipe to process 2 P2 = multiprocessing. Process (target = proc2, args = (pipe [1],)     P1.start () P2.start () P1.join () P2.join ()

Not only the pipe of multiprocessing, but also other pipe implementations are just the play between two processes. I will give it to you, you will receive it or you will, and I will receive it. Of course, it can also be made into a duplex state.

In the case of queue, more processes can be involved. The usage is similar to that of some other queue.


See the documents on the official website:

Multiprocessing. Pipe ([duplex])

Returns a pair (conn1, conn2) of Connection objects representing the ends of a pipe.

# Two pipe objects. Use these two objects to communicate with each other.


If duplex is True (the default) then the pipe is bidirectional. if duplex is False then the pipe is unidirectional: conn1 can only be used for processing messages and conn2 can only be used for sending messages.


Class multiprocessing. Queue ([maxsize])

Returns a process shared queue implemented using a pipe and a few locks/semaphores. when a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.

# Maximum number of queues


The usual Queue. Empty and Queue. Full exceptions from the standard library's Queue module are raised to signal timeouts.


Queue implements all the methods of Queue. Queue has t for task_done () and join ().


Qsize ()

Return the approximate size of the queue. Because of multithreading/multiprocessing semantics, this number is not reliable.

# Queue size


Note that this may raise NotImplementedError on Unix platforms like Mac OS X where sem_getvalue () is not implemented.


Empty ()

Return True if the queue is empty, False otherwise. Because of multithreading/multiprocessing semantics, this is not reliable.

# Whether the hole is saved. If it is null, return a True state.


Full ()

Return True if the queue is full, False otherwise. Because of multithreading/multiprocessing semantics, this is not reliable.

# Whether the queue status is full.


Put (obj [, block [, timeout])

Put obj into the queue. if the optional argument block is True (the default) and timeout is None (the default), block if necessary until a free slot is available. if timeout is a positive number, it blocks at most timeout seconds and raises the Queue. full exception if no free slot was available within that time. otherwise (block is False), put an item on the queue if a free slot is immediately available, else raise the Queue. full exception (timeout is ignored in that case ).

# Add a timeout value to the queue.

Put_nowait (obj)

Equivalent to put (obj, False ).

# This is not blocked.


Get ([block [, timeout])

Remove and return an item from the queue. if optional args block is True (the default) and timeout is None (the default), block if necessary until an item is available. if timeout is a positive number, it blocks at most timeout seconds and raises the Queue. empty exception if no item was available within that time. otherwise (block is False), return an item if one is immediately available, else raise the Queue. empty exception (timeout is ignored in that case ).

# Obtaining status


Get_nowait ()

Equivalent to get (False ).

# Data in a non-congested get queue


Queue has a few additional methods not found in Queue. Queue. These methods are usually unnecessary for most code:


Close ()

Indicate that no more data will be put on this queue by the current process. the background thread will quit once it has flushed all buffered data to the pipe. this is called automatically when the queue is garbage collected.

# Close to save the resources of the current process.

 

I configured the length of the multiprocessing team to be three. When I put the fourth one, I will find that one of them is blocked. He is waiting, and someone will get one of the data, at that time, he can continue to plug in. If put_nowait () is used, an error occurs immediately when the queue exceeds the threshold.


/System/Library/Frameworks/Python. framework/Versions/2.7/lib/python2.7/multiprocessing/queues. pyc in put_nowait (self, obj)


/System/Library/Frameworks/Python. framework/Versions/2.7/lib/python2.7/multiprocessing/queues. pyc in put (self, obj, block, timeout)




The following is a piece of test code. You can run the demo and feel it.
   

The code is as follows:

Copy code

# Coding: UTF-8 Import OS Import multiprocessing Import time # Write to worker Def inputQ (queue ): While True: Info = "process Number % s: time: % s" % (OS. getpid (), int (time. time ())) Queue. put (info) Time. sleep (1) # Getting worker Def outputQ (queue, lock ): While True: Info = queue. get () # Lock. acquire () Print (str (OS. getpid () + '(get):' + info) # Lock. release () Time. sleep (1) #============================== # Main Record1 = [] # store input processes Record2 = [] # store output processes Lock = multiprocessing. Lock () # To prevent messy print Queue = multiprocessing. Queue (3)   # Input processes For I in range (10 ): Process = multiprocessing. Process (target = inputQ, args = (queue ,)) Process. start () Record1.append (process)   # Output processes For I in range (10 ): Process = multiprocessing. Process (target = outputQ, args = (queue, lock )) Process. start () Record2.append (process)

Well, I briefly talked about the usage of pipe and queue. Actually, I tried to pull python pipe today. As a result, google searched and saw the pipe of multiprocessing. After writing pipe, I felt that there was too little content in the article, so I added the queue...