Python multi-process communication queue, Pipe, Value, array instance

The difference between a queue and a pipe: pipe is used to communicate between two processes. Queue is used to communicate between multiple processes. These two methods are the basic method for multi-process communication for all systems, and almost all languages support both methods.

1) Queue & Joinablequeue

Queue is used to pass messages between processes, and any object that can be pickle-able can be added to the queue.

Multiprocessing. Joinablequeue is a subclass of the queue, with the addition of the Task_done () and join () methods.

Task_done () is used to tell the queue that a task is complete. In general, get a task in call get (), call Task_done () after the task ends to notify the queue that the current task is complete.

Join () blocks until all the tasks in the queue are processed (that is, the Task_done method is called).

Code:

Copy the Code code as follows:

Import multiprocessing
Import time

Class Consumer (multiprocessing. Process):

def __init__ (self, Task_queue, result_queue):
Multiprocessing. Process.__init__ (self)
Self.task_queue = Task_queue
Self.result_queue = Result_queue

def run (self):
Proc_name = Self.name
While True:
Next_task = Self.task_queue.get ()
If Next_task is None:
# Poison pill means shutdown
Print ('%s:exiting '% proc_name)
Self.task_queue.task_done ()
Break
Print ('%s:%s '% (Proc_name, next_task))
Answer = Next_task () # __call__ ()
Self.task_queue.task_done ()
Self.result_queue.put (Answer)
Return

Class Task (object):
def __init__ (self, A, b):
SELF.A = A
SELF.B = b
def __call__ (self):
Time.sleep (0.1) # Pretend to take some time-to-do
Return '%s *%s =%s '% (SELF.A, self.b, SELF.A * self.b)
def __str__ (self):
Return '%s *%s '% (SELF.A, self.b)

if __name__ = = ' __main__ ':
# Establish communication queues
tasks = multiprocessing. Joinablequeue ()
Results = multiprocessing. Queue ()

# Start Consumers
Num_consumers = Multiprocessing.cpu_count ()
Print (' Creating%d consumers '% num_consumers)
Consumers = [Consumer (Tasks, results)
For I in range (num_consumers)]
For W in consumers:
W.start ()

# Enqueue Jobs
Num_jobs = 10
For I in Range (num_jobs):
Tasks.put (Task (i, i))

# Add A poison pill for each consumer
For I in Range (num_consumers):
Tasks.put (None)

# Wait for any of the tasks to finish
Tasks.join ()

# Start Printing Results
While Num_jobs:
result = Results.get ()
Print (' Result: ', result)
Num_jobs-= 1

Note tip: Use None to indicate that the task has finished processing.

Operation Result:

2) Pipe

Pipe () returns a pair of connection objects that represent both ends of the pipe. Each object has the Send () and recv () methods.

Code:
Copy the Code code as follows:

From multiprocessing import Process, Pipe

def f (conn):
Conn.send ([All, None, ' hello '])
Conn.close ()

if __name__ = = ' __main__ ':
Parent_conn, Child_conn = Pipe ()
p = Process (Target=f, args= (Child_conn,))
P.start ()
P.join ()
Print (Parent_conn.recv ()) # prints "[All, None, ' hello ']"

3) Value + Array

The Value + Array is a method for sharing memory-mapped files in Python, which is faster.

Copy the Code code as follows:

From multiprocessing import Process, Value, Array

def f (N, a):
N.value = N.value + 1
For I in range (Len (a)):
A[i] = a[i] * 10

if __name__ = = ' __main__ ':
num = Value (' i ', 1)
arr = Array (' I ', Range (10))

p = Process (target=f, args= (num, arr))
P.start ()
P.join ()

Print (Num.value)
Print (arr[:])

P2 = Process (target=f, args= (num, arr))
P2.start ()
P2.join ()

Print (Num.value)
Print (arr[:])

# The output is:
# 2
# [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
# 3
# [0, 100, 200, 300, 400, 500, 600, 700, 800, 900]

