A multi-tasking can use one process as the master to assign tasks, and other processes to perform tasks as a worker.
This allows master to be placed on a single computer, workers on other computers for distributed processes.
#taskmanager .py#!/usr/bin/env pythonimport random, time, queuefrom Multiprocessing.managers import basemanagertask_queue = queue.queue () result_queue = queue.queue () Class queuemanager (Basemanager): passqueuemanager.register (' Get _task_queue ', callable=lambda: task_queue) queuemanager.register (' Get_result_queue ', callable= Lambda: result_queue) Manager = queuemanager (address= (" 5000"), authkey= ' abc ') Manager.start () Task = manager.get_task_queue () Result = manager.get_result_queue () for I in range (): n = random.randint (0, 10000) print (' put task %d ... ' % n) task.put (n) print (' Try get results ') for i in range: r = result.get ( timeout=10) print (' Result: %s ' %r) Manager.shutdown ()
Task_queue and Result_queue are two queues, each storing tasks and results. They are used to communicate between processes, exchanging objects.
The official web site has the following examples.
From multiprocessing import Process, queuedef F (queue): Queue.put ([All, None, ' hello ']) if __name__ = = ' __main__ ': Q = Queue () p = Process (target=f, args= (Q,)) P.start () print q.get () # prints "[All, None, ' hello ']" p.join ()
where P is a process, and there is a queue Q for the main process. list [All, None, ' hello '] is passed from the P process to the main process.
Because it is a distributed environment, the data placed in the queue needs to wait for the workers machine to be processed before it is read, so it needs to encapsulate the queue with QueueManager into the network. This is done by
Queuemanager.register (' Get_task_queue ', callable=lambda:task_queue)
Implemented. We give Task_queue the network call interface takes a name Get_task_queue, and result_queue name is get_result_queue, convenient to distinguish which queue to operate.
Task.put (n) is writing data to Task_queue, which is equivalent to assigning tasks. Result.get () is the result of waiting for workers processing to return.
The following is the code for workers.
#task_worker .py#!/usr/bin/env pythonimport time, sys, queuefrom Multiprocessing.managers import basemanagerclass queuemanager (BaseManager): passqueuemanager.register (' Get_task_queue ') queuemanager.register (' Get_result_queue ') server_addr = ' 127.0.0.1 ' Print (' connect to server %s ... ' % server_addr) m = QueueManager (address= (server_addr, 5000), authkey= ' abc ') M.connect () Task = m.get_task_queue () Result = m.get_result_queue () For i in range (Ten): try: n = task.get (timeout=1) print (' run task %d * %d ... ' % (n, n)) r = '%d * %d = %d ' % (n, n, n* N) tIme.sleep (1) result.put (R) except Queue.empty: print (' Task queue is empty. ') Print (' Worker exit. ')
Task_worker the name of the QueueManager registered here must be the same as in Task_manager. Note the taskworker.py taskmanager.py in progress. Compare the above example, you can see queue object is passed over the network from another process. Just here the delivery and network communication by queuemanager completed.
The main function of Task_worker is to take out the data allocated in the Task_queue for the square operation and then put it into result_queue. So Task_manager can get the result of the calculation.
The results are as follows:
References
[1].http://blog.csdn.net/fireroll/article/details/38895485
[2].http://www.jb51.net/article/58004.htm
[3].https://docs.python.org/3/library/multiprocessing.html?highlight=queuemanager
Python Distributed Process