Python Distributed Process

Distributed processes

If a multi-process program with queue communication is already running on the same machine,
You want to distribute the process of sending tasks and the process of processing tasks to two machines.

By exposing the queue through the network through the Managers module, you can let other machines ' processes access the queue.

The service process is responsible for starting the queue, registering the queue on the network, and then writing the task to the queue

# task_master.py

Import random, time, queue
From multiprocessing.managers import Basemanager

# Send queue for task:
Task_queue = queue. Queue ()
# Queue to receive results:
Result_queue = queue. Queue ()

# QueueManager inherited from Basemanager:
Class QueueManager (Basemanager):
Pass

# Two queues are registered on the network, the callable parameter is associated with the queue object:

Queuemanager.register (' Get_task_queue ', callable=lambda:task_queue)
Queuemanager.register (' Get_result_queue ', callable=lambda:result_queue)

# Bind Port 5000, set CAPTCHA ' abc ':

Manager = QueueManager (address= (","), Authkey=b ' abc ')

# Start Queue:

Manager.start ()

# Get a Queue object accessed over the network:

task = Manager.get_task_queue ()
result = Manager.get_result_queue ()

# Put a few quests in:

For I in range (10):
n = random.randint (0, 10000)
Print (' Put task%d ... '% n)
Task.put (N)

# read results from the result queue:

Print (' Try get results ... ')
For I in range (10):
R = Result.get (timeout=10)
Print (' Result:%s '% r)
Close
Manager.shutdown ()
Print (' Master exit. ')

The queue interface that must be obtained through Manager.get_task_queue () is added

Then, start the task process on another machine (it can also be started on this machine)

# task_worker.py

Import time, sys, queue
From multiprocessing.managers import Basemanager

# Create a similar QueueManager:
Class QueueManager (Basemanager):
Pass

# Since this queuemanager only gets the queue from the network, it only provides the name when registering:
Queuemanager.register (' Get_task_queue ')
Queuemanager.register (' Get_result_queue ')

# Connect to the server, which is the machine running task_master.py:
server_addr = ' 127.0.0.1 '
print (' Connect to server%s ... '% server_addr)
# port and Authenticode note remain exactly the same as the task_master.py settings:
m = QueueManager (address= (server_addr, mm), authkey=b ' abc ')
# from the network connection:
M.connect ()
# Gets the object of the queue:
task = M.get_task_queue ()
result = M.get_result_queue ()
# Pull tasks from the task queue and write the results to the result queue:
For I in Range (Ten):
Try:
n = task.get (timeout=1)
print (' Run task%d *%d ... '% (n, N))
r = '%d *%d =%d '% (n, N, n*n)
time.sleep (1)
Result.put (R)
except Queue.empty:
print (' Task queue is empty. ')
# processing Ends:
print (' Worker exit. ')

The task process is to connect to the service process over the network, so specify the IP of the service process.

Start the task_master.py service process first:

$ python3 task_master.py
put task 3411 ...
put task 1605 ...
put task 1398 ...
put task 4729 ...
put task 5300 ...
put task 7471 ...
put task ...
put task 4219 ...
put task 339 ...
put task 7866 ...
try get results ...

After the task_master.py process sends the task, it begins to wait for the result queue. Now start the task_worker.py process:

$ python3 task_worker.py
Connect to server 127.0.0.1 ...
Run Task 3411 * 3411 ...
Run Task 1605 * 1605 ...
Run Task 1398 * 1398 ...
Run Task 4729 * 4729 ...
Run Task 5300 * 5300 ...
Run Task 7471 * 7471 ...
Run Task ...
Run Task 4219 * 4219 ...
Run Task 339 * 339 ...
Run Task 7866 * 7866 ...
worker exit.

The task_worker.py process ends, and the results will continue to print in the task_master.py process:

result:3411 * 3411 = 11634921
result:1605 * 1605 = 2576025
result:1398 * 1398 = 1954404
result:4729 * 4729 = 22363441
result:5300 * 5300 = 28090000
result:7471 * 7471 = 55815841
result:68 * 68 = 4624
result:4219 * 4219 = 17799961
result:339 * 339 = 114921
result:7866 * 7866 = 61873956

The queue object is stored in the task_master.py process

The reason that a queue can be accessed through the network is through QueueManager.

Python Distributed Process