Python uses multiprocessing to implement a simple Distributed Job Scheduling System

Mutilprocess manages processes like threads. This is the core of mutilprocess. It is very similar to threading, and the utilization of multi-core CPU is much better than threading, this article describes how to use multiprocessing to implement a simple Distributed Job Scheduling System in Python. If you need to use multiprocessing to manage processes like threads, you can refer to mutilprocess. This is the core of mutilprocess, similar to threading, the CPU utilization for multiple cores is much better than threading.

Introduction

The multiprocessing module of Python not only supports multi-process, but also supports multi-process distribution on multiple machines. A service process can be used as a scheduler to distribute tasks to multiple processes on multiple other machines and rely on network communication.

Think of this, just wondering if this module can be used to implement a simple job scheduling system.

Implementation

Job

First, create a Job class. To make the test simple, only one job id attribute is included.

Job. py

#!/usr/bin/env python# -*- coding: utf-8 -*-class Job:def __init__(self, job_id):self.job_id = job_id

Master

The Master is used to distribute jobs and display information about Jobs completed

Master. py

#!/usr/bin/env python# -*- coding: utf-8 -*-from Queue import Queuefrom multiprocessing.managers import BaseManagerfrom job import Job

Class Master:

Def _ init _ (self): # self. dispatched_job_queue = Queue () # completed Job Queue self. finished_job_queue = Queue () def get_dispatched_job_queue (self): return self. dispatched_job_queuedef get_finished_job_queue (self): return self. finished_job_queuedef start (self): # register the dispatch job queue and complete the job queue to the network BaseManager. register ('get _ dispatched_job_queue ', callable = self. get_dispatched_job_queue) BaseManager. register ('get _ finished_job_queue ', callable = self. get_finished_job_queue) # listening port and starting Service manager = BaseManager (address = ('0. 0.0.0 ', 8888), authkey = 'job') manager. start () # Use the method registered above to obtain the queue dispatched_jobs = manager. get_dispatched_job_queue () finished_jobs = manager. get_finished_job_queue () # distribute 10 jobs at a time. After all 10 jobs are run, send another 10 jobs, job_id = 0 while True: for I in range (0, 10): job_id = job_id + 1job = Job (job_id) print ('dispatch job: % s' % job. job_id) dispatched_jobs.put (job) while not dispatched_jobs.empty (): job = finished_jobs.get (60) print ('finished Job: % s' % job. job_id) manager. shutdown () if _ name _ = "_ main _": master = Master () master. start ()

Slave

Slave is used to run the jobs distributed by the master and return the results

Slave. py

#!/usr/bin/env python# -*- coding: utf-8 -*-import timefrom Queue import Queuefrom multiprocessing.managers import BaseManagerfrom job import Job

Class Slave:

Def _ init _ (self): # self. dispatched_job_queue = Queue () # self. finished_job_queue = Queue ()

Def start (self ):

# Register the dispatch job queue and complete job queue on the network BaseManager. register ('get _ dispatched_job_queue ') BaseManager. register ('get _ finished_job_queue ') # connect to masterserver = '2017. 0.0.1 'print ('connect to server % s... '% server) manager = BaseManager (address = (server, 8888), authkey = 'job') manager. connect () # Use the method registered above to obtain the queue dispatched_jobs = manager. get_dispatched_job_queue () finished_jobs = manager. get_finished_job_queue () # Run the job and return the result. Here, the job is simulated, so the returned job is while True: job = dispatched_jobs.get (timeout = 1) print ('run job: % s' % job. job_id) time. sleep (1) finished_jobs.put (job) if _ name _ = "_ main _": slave = Slave () slave. start ()

Test

Open three linux terminals, run master on the first terminal, and run slave on the second and third terminals respectively. The running result is as follows:

Master

$ python master.py Dispatch job: 1Dispatch job: 2Dispatch job: 3Dispatch job: 4Dispatch job: 5Dispatch job: 6Dispatch job: 7Dispatch job: 8Dispatch job: 9Dispatch job: 10Finished Job: 1Finished Job: 2Finished Job: 3Finished Job: 4Finished Job: 5Finished Job: 6Finished Job: 7Finished Job: 8Finished Job: 9Dispatch job: 11Dispatch job: 12Dispatch job: 13Dispatch job: 14Dispatch job: 15Dispatch job: 16Dispatch job: 17Dispatch job: 18Dispatch job: 19Dispatch job: 20Finished Job: 10Finished Job: 11Finished Job: 12Finished Job: 13Finished Job: 14Finished Job: 15Finished Job: 16Finished Job: 17Finished Job: 18Dispatch job: 21Dispatch job: 22Dispatch job: 23Dispatch job: 24Dispatch job: 25Dispatch job: 26Dispatch job: 27Dispatch job: 28Dispatch job: 29Dispatch job: 30

Slave1

$ python slave.py Connect to server 127.0.0.1...Run job: 1 Run job: 2 Run job: 3 Run job: 5 Run job: 7 Run job: 9 Run job: 11 Run job: 13 Run job: 15 Run job: 17 Run job: 19 Run job: 21 Run job: 23 

Slave2

$ python slave.py Connect to server 127.0.0.1...Run job: 4 Run job: 6 Run job: 8 Run job: 10 Run job: 12 Run job: 14 Run job: 16 Run job: 18 Run job: 20 Run job: 22 Run job: 24 

The above content is a simple Distributed Job Scheduling System Using multiprocessing!