The difference between queue and pipe: pipe is used to communicate between two processes. The queue is used to implement communication between multiple processes. These two methods are the basic method for all systems to communicate in multiple processes, and almost all languages support both of these methods.
1) Queue & Joinablequeue
The queue is used to pass messages between processes, and any object that can be pickle-able can be added to the queue.
Multiprocessing. Joinablequeue is a subclass of the queue, adding the Task_done () and join () methods.
Task_done () is used to tell the queue that a task completes. Generally, a task is obtained by calling get (), and Task_done () is called after the task ends to notify the queue that the current task is complete.
Join () blocks until all tasks in the queue are processed (that is, the Task_done method is invoked).
Code:
Copy Code code as follows:
Import multiprocessing
Import time
Class Consumer (multiprocessing. Process):
def __init__ (self, Task_queue, result_queue):
Multiprocessing. Process.__init__ (self)
Self.task_queue = Task_queue
Self.result_queue = Result_queue
def run (self):
Proc_name = Self.name
While True:
Next_task = Self.task_queue.get ()
If Next_task is None:
# Poison pill means shutdown
Print ('%s:exiting '% proc_name)
Self.task_queue.task_done ()
Break
Print ('%s:%s '% (Proc_name, next_task))
Answer = Next_task () # __call__ ()
Self.task_queue.task_done ()
Self.result_queue.put (Answer)
Return
Class Task (object):
def __init__ (self, A, b):
SELF.A = A
SELF.B = b
def __call__ (self):
Time.sleep (0.1) # Pretend to take some
Return '%s *%s =%s '% (SELF.A, self.b, SELF.A * self.b)
def __str__ (self):
Return '%s *%s '% (SELF.A, self.b)
if __name__ = = ' __main__ ':
# Establish communication queues
tasks = multiprocessing. Joinablequeue ()
Results = multiprocessing. Queue ()
# Start Consumers
Num_consumers = Multiprocessing.cpu_count ()
Print (' creating%d consumers '% num_consumers)
Consumers = [Consumer (Tasks, results)
For I in range (num_consumers)]
For W in consumers:
W.start ()
# Enqueue Jobs
Num_jobs = 10
For I in Range (num_jobs):
Tasks.put (Task (i, i))
# Add A poison pill for each consumer
For I in Range (num_consumers):
Tasks.put (None)
# Wait for all ' the tasks to finish
Tasks.join ()
# Start Printing Results
While Num_jobs:
result = Results.get ()
Print (' Result: ', result)
Num_jobs-= 1
Note tip: Use None to indicate that task processing is complete.
Run Result:
2) Pipe
Pipe () returns a pair of connected objects, representing the ends of the pipe. Each object has a send () and recv () method.
Code:
Copy Code code as follows:
From multiprocessing import Process, Pipe
def f (conn):
Conn.send ([, None, ' hello '])
Conn.close ()
if __name__ = = ' __main__ ':
Parent_conn, Child_conn = Pipe ()
p = Process (Target=f, args= (Child_conn,))
P.start ()
P.join ()
Print (Parent_conn.recv ()) # prints "[A, None, ' hello ']"
3) Value + Array
The Value + Array is a way to share memory-mapped files in Python faster.
Copy Code code as follows:
From multiprocessing import Process, Value, Array
def f (N, a):
N.value = N.value + 1
For I in range (Len (a)):
A[i] = a[i] * 10
if __name__ = = ' __main__ ':
num = Value (' i ', 1)
arr = Array (' I ', Range (10))
p = Process (target=f, args= (num, arr))
P.start ()
P.join ()
Print (Num.value)
Print (arr[:])
P2 = Process (target=f, args= (num, arr))
P2.start ()
P2.join ()
Print (Num.value)
Print (arr[:])
# The output is:
# 2
# [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
# 3
# [0, 100, 200, 300, 400, 500, 600, 700, 800, 900]