Python class library 32 [multi-process communication queue + pipe + value + array]

Multi-process communication

The difference between queue and pipe: pipe is used to communicate between two processes. Queue is used to implement communication between multiple processes. These two methods are the basic methods for multi-process communication in all systems. Almost all languages support these two methods.

 

1) queue & joinablequeue

Queue is used to transmit messages between processes. Any object that can be pickle-Able can be added to queue.

Multiprocessing. joinablequeue is a subclass of queue. The task_done () and join () methods are added.

 

Task_done () is used to tell the queue that a task is completed. Generally, get a task by calling get () and call task_done () after the task ends to notify the queue of the completion of the current task.

Join () is blocked until all tasks in the queue are processed (that is, the task_done method is called ).

Code:

Import Multiprocessing
Import Time

ClassConsumer (multiprocessing. Process ):

Def _ Init __(Self, task_queue, result_queue ):
Multiprocessing. process._ Init __(Self)
Self. task_queue = task_queue
Self. result_queue = result_queue

Def Run (Self ):
Proc_name = self. Name
While True:
Next_task = self. task_queue.get ()
If Next_task Is None:
# Poison pill means Shutdown
Print ( ' % S: exiting ' % Proc_name)
Self. task_queue.task_done ()
Break
Print ( ' % S: % s ' % (Proc_name, next_task ))
Answer = next_task () # _ Call __()
Self. task_queue.task_done ()
Self. result_queue.put (answer)
Return

Class Task (object ):
Def _ Init __ (Self, a, B ):
Self. A =
Self. B = B
Def _ Call __ (Self ):
Time. Sleep (0.1) # Pretend to take some time to do the work
Return ' % S * % s = % s ' % (Self. A, self. B, self. A * Self. B)
Def _ STR __ (Self ):
Return ' % S * % s ' % (Self. A, self. B)

If _ Name __ =' _ Main __ ' :
# Establish communication queues
Tasks = multiprocessing. joinablequeue ()
Results = multiprocessing. Queue ()

# Start consumers
Num_consumers = multiprocessing. cpu_count ()
Print ( ' Creating % d consumers ' % Num_consumers)
Consumers = [consumer (tasks, Results)
For I In Range (num_consumers)]
For W In Consumers:
W. Start ()

# Enqueue jobs
Num_jobs = 10
For I In Range (num_jobs ):
Tasks. Put (task (I, I ))

# Add a poison pill for each consumer
For I In Range (num_consumers ):
Tasks. Put (none)

#Wait for all of the tasks to finish
Tasks. Join ()

#Start printing results
WhileNum_jobs:
Result = results. Get ()
Print('Result:', Result)
Num_jobs-= 1

Note tips: use none to indicate that the task has been processed.

 

Running result:

 

2) Pipe

 

Pipe () returns a pair of connection objects, representing the two ends of pipe. Each object has the send () and Recv () methods.

 

 

Code:

 

From Multiprocessing Import Process, pipe

DefF (conn ):
Conn. Send ([42, none,'Hello'])
Conn. Close ()

If _ Name __='_ Main __':
Parent_conn, child_conn = pipe ()
P = process (target = F, argS = (child_conn ,))
P. Start ()
P. Join ()
Print(Parent_conn.recv ())#Prints "[42, none, 'Hello']"

 

 

3) value + Array

Value + array is a method for sharing memory ing files in Python, which is faster.

From Multiprocessing Import Process, value, array

DefF (n, ):
N. value = n. Value + 1
ForIInRange (LEN ()):
A [I] = A [I] * 10

If _ Name __='_ Main __':
Num = value ('I', 1)
Arr = array ('I', Range (10 ))

P = process (target = F, argS = (Num, arr ))
P. Start ()
P. Join ()

Print(Num. value)
Print(ARR [:])

P2 = process (target = F, argS = (Num, arr ))
P2.start ()
P2.join ()

Print(Num. value)
Print(ARR [:])

#The output is:
#2
#[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
#3
#[0,100,200,300,400,500,600,700,800,900]

Refer:
The Python standard library by example

Http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html

 

Complete!