Python multi-process communication Queue, Pipe, Value, Array instance

This article mainly introduces the Python multi-process communication Queue, Pipe, Value, Array instance, queue and pipe are used to transmit messages between processes. Value + Array is a method for sharing memory ing files in python. For more information, see the difference between queue and pipe: pipe is used for communication between two processes. Queue is used to implement communication between multiple processes. These two methods are the basic methods for multi-process communication in all systems. almost all languages support these two methods.

1) Queue & JoinableQueue

Queue is used to transmit messages between processes. any object that can be pickle-able can be added to queue.

Multiprocessing. JoinableQueue is a subclass of Queue. the task_done () and join () methods are added.

Task_done () is used to tell the queue that a task is completed. Generally, get a task by calling get () and call task_done () after the task ends to notify the Queue of the completion of the current task.

Join () is blocked until all tasks in the queue are processed (that is, the task_done method is called ).

Code:

The code is as follows:

Import multiprocessing
Import time

Class Consumer (multiprocessing. Process ):

Def _ init _ (self, task_queue, result_queue ):
Multiprocessing. Process. _ init _ (self)
Self. task_queue = task_queue
Self. result_queue = result_queue

Def run (self ):
Proc_name = self. name
While True:
Next_task = self. task_queue.get ()
If next_task is None:
# Poison pill means shutdown
Print ('% s: Exiting' % proc_name)
Self. task_queue.task_done ()
Break
Print ('% s: % s' % (proc_name, next_task ))
Answer = next_task () # _ call __()
Self. task_queue.task_done ()
Self. result_queue.put (answer)
Return

Class Task (object ):
Def _ init _ (self, a, B ):
Self. a =
Self. B = B
Def _ call _ (self ):
Time. sleep (0.1) # pretend to take some time to do the work
Return '% s * % s = % s' % (self. a, self. B, self. a * self. B)
Def _ str _ (self ):
Return '% s * % s' % (self. a, self. B)

If _ name _ = '_ main __':
# Establish communication queues
Tasks = multiprocessing. JoinableQueue ()
Results = multiprocessing. Queue ()

# Start consumers
Num_consumers = multiprocessing. cpu_count ()
Print ('creating % d consumers' % num_consumers)
Consumers = [Consumer (tasks, results)
For I in range (num_consumers)]
For w in consumers:
W. start ()

# Enqueue jobs
Num_jobs = 10
For I in range (num_jobs ):
Tasks. put (Task (I, I ))

# Add a poison pill for each consumer
For I in range (num_consumers ):
Tasks. put (None)

# Wait for all of the tasks to finish
Tasks. join ()

# Start printing results
While num_jobs:
Result = results. get ()
Print ('result: ', Result)
Num_jobs-= 1

Note tips: Use None to indicate that the task has been processed.

Running result:

2) pipe

Pipe () returns a pair of connection objects, representing the two ends of pipe. Each object has the send () and recv () methods.

Code:

The code is as follows:

From multiprocessing import Process, Pipe

Def f (conn ):
Conn. send ([42, None, 'Hello'])
Conn. close ()

If _ name _ = '_ main __':
Parent_conn, child_conn = Pipe ()
P = Process (target = f, args = (child_conn ,))
P. start ()
P. join ()
Print (parent_conn.recv () # prints "[42, None, 'Hello']"

3) Value + Array

Value + Array is a method for sharing memory ing files in python, which is faster.

The code is as follows:

From multiprocessing import Process, Value, Array

Def f (n, ):
N. value = n. value + 1
For I in range (len ()):
A [I] = a [I] * 10

If _ name _ = '_ main __':
Num = Value ('I', 1)
Arr = Array ('I', range (10 ))

P = Process (target = f, args = (num, arr ))
P. start ()
P. join ()

Print (num. value)
Print (arr [:])

P2 = Process (target = f, args = (num, arr ))
P2.start ()
P2.join ()

Print (num. value)
Print (arr [:])

# The output is:
#2
# [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]
#3
# [0,100,200,300,400,500,600,700,800,900]