Python distributed inter-process communication

Source: Internet
Author: User

In thread and process, the process should be preferred because the process is more stable, and the process can be distributed across multiple machines, and the thread can only be distributed to multiple CPUs on the same machine.

multiprocessing module not only supports multiple processes, where managers sub-module also supports the distribution of multiple processes to multiple machines. A service process can act as a dispatcher, distributing tasks across multiple processes and relying on network traffic. Because managers module is well-encapsulated, and it is easy to write distributed multi-process programs without having to understand the details of network traffic.

For example, if we already have a Queue multi-process program running on the same machine through communication, now, due to the heavy workload of processing tasks, we want to distribute the process of sending tasks and the process of processing tasks to two machines. How to implement with distributed process?

The originalQueueCan continue to be used, however, bymanagersModule toQueueExposed through the network, you can let the process of other machines accessQueueThe


Mast end:

#taskmanager .py#!/usr/bin/env pythonimport random, time, queuefrom  Multiprocessing.managers import basemanager task_queue = queue.queue () Result_queue  = queue.queue ()  class queuemanager (Basemanager):    pass  Queuemanager.register (' Get_task_queue ',  callable=lambda: task_queue) queuemanager.register (' get_ Result_queue ',  callable=lambda: result_queue)  manager = queuemanager (address= (' ',  5000),  authkey= ' abc ') Manager.start () task = manager.get_task_queue () result  = manager.get_result_queue ()  for i in range (Ten):    n  = random.randint (0, 10000)     print (' put task %d ... '  % n)     task.put (n) print (' Try get results ... ')   for i in  Range:     r = result.get (timeout=10)     print (' result: %s '  %r) Manager.shutdown () 

Slave end

#task_worker .py#!/usr/bin/env pythonimport time, sys, queuefrom  Multiprocessing.managers import basemanager class queuemanager (BaseManager):     pass queuemanager.register (' Get_task_queue ') queuemanager.register (' Get_result_queue ')   server_addr =  ' ' Print (' connect to server %s ... '  % server_addr) m  = queuemanager (address= (server_addr, 5000),  authkey= ' abc ') M.connect () task =  M.get_task_queue () Result = m.get_result_queue ()  for i in range (Ten):     try:        n = task.get (timeout=1)          print (' run task %d * %d ... '  %  (n, n))         r =  '%d * %d = %d '  %  ( N, n, n*n)     &Nbsp;   time.sleep (1)         result.put (R)      except queue.empty:        print (' Task queue  is empty. ') Print (' Worker exit. ')

This article from "for The Good Life" blog, declined reprint!

Python distributed inter-process communication

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.