Python distributed inter-process communication

In thread and process, the process should be preferred because the process is more stable, and the process can be distributed across multiple machines, and the thread can only be distributed to multiple CPUs on the same machine.

multiprocessing module not only supports multiple processes, where managers sub-module also supports the distribution of multiple processes to multiple machines. A service process can act as a dispatcher, distributing tasks across multiple processes and relying on network traffic. Because managers module is well-encapsulated, and it is easy to write distributed multi-process programs without having to understand the details of network traffic.

For example, if we already have a Queue multi-process program running on the same machine through communication, now, due to the heavy workload of processing tasks, we want to distribute the process of sending tasks and the process of processing tasks to two machines. How to implement with distributed process?

The originalQueueCan continue to be used, however, bymanagersModule toQueueExposed through the network, you can let the process of other machines accessQueueThe

Python27

Mast end:

#taskmanager .py#!/usr/bin/env pythonimport random, time, queuefrom  Multiprocessing.managers import basemanager task_queue = queue.queue () Result_queue  = queue.queue ()  class queuemanager (Basemanager):    pass  Queuemanager.register (' Get_task_queue ',  callable=lambda: task_queue) queuemanager.register (' get_ Result_queue ',  callable=lambda: result_queue)  manager = queuemanager (address= (' 127.0.0.1 ',  5000),  authkey= ' abc ') Manager.start () task = manager.get_task_queue () result  = manager.get_result_queue ()  for i in range (Ten):    n  = random.randint (0, 10000)     print (' put task %d ... '  % n)     task.put (n) print (' Try get results ... ')   for i in  Range:     r = result.get (timeout=10)     print (' result: %s '  %r) Manager.shutdown () 

Slave end

#task_worker .py#!/usr/bin/env pythonimport time, sys, queuefrom  Multiprocessing.managers import basemanager class queuemanager (BaseManager):     pass queuemanager.register (' Get_task_queue ') queuemanager.register (' Get_result_queue ')   server_addr =  ' 127.0.0.1 ' Print (' connect to server %s ... '  % server_addr) m  = queuemanager (address= (server_addr, 5000),  authkey= ' abc ') M.connect () task =  M.get_task_queue () Result = m.get_result_queue ()  for i in range (Ten):     try:        n = task.get (timeout=1)          print (' run task %d * %d ... '  %  (n, n))         r =  '%d * %d = %d '  %  ( N, n, n*n)     &Nbsp;   time.sleep (1)         result.put (R)      except queue.empty:        print (' Task queue  is empty. ') Print (' Worker exit. ')

This article from "for The Good Life" blog, declined reprint!

Python distributed inter-process communication