Python uses Redis to implement a simple job scheduling system

Python uses Redis to implement a simple job scheduling system
Python uses Redis to implement a simple job scheduling system

As a typical example of a memory database, Redis has been used in many application scenarios, here, we only talk about how to use the pub/sub function of Redis to implement a simple job scheduling system. I just want to present a simple idea here, so there are still a lot of things to consider that are not included in this example, such as error processing and persistence.

The following are Implementation ideas:

MyMaster: the master node program of the cluster, which is responsible for generating jobs, distributing jobs, and obtaining execution results. MySlave: The computing node program of the cluster. Each computing node has one. It obtains and runs the job and sends the result to the master node. Channel CHANNEL_DISPATCH: each slave node subscribes to a channel, for example, "CHANNEL_DISPATCH _ [idx or machine name]". The master will dispatch the publish job to this channel. Channel CHANNEL_RESULT: the channel used to save the Job results. The master and slave share this channel. The master subscribes to this channel to obtain the job running results. Each slave is responsible for publishing the job execution results to this channel. Master code

#!/usr/bin/env python# -*- coding: utf-8 -*-import timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MyMaster():    def __init__(self):        pass    def start(self):        MyServerResultHandleThread().start()        MyServerDispatchThread().start()class MyServerDispatchThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        for i in range(1, 100):            channel = CHANNEL_DISPATCH + '_' + str(random.randint(1, 3))            print("Dispatch job %s to %s" % (str(i), channel))            ret = r.publish(channel, str(i))            if ret == 0:                print("Dispatch job %s failed." % str(i))            time.sleep(5)class MyServerResultHandleThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        p = r.pubsub()        p.subscribe(CHANNEL_RESULT)        for message in p.listen():            if message['type'] != 'message':                continue            print("Received finished job %s" % message['data'])if __name__ == "__main__":    MyMaster().start()    time.sleep(10000)

Description

MyMaster class-master main program, used to start dispatch and resulthandler threads MyServerDispatchThread class-dispatch job threads, generate jobs and distribute them to the compute node MyServerResultHandleThread class-job running result processing thread, obtain the job result from the channel and display the Slave code.

#!/usr/bin/env python# -*- coding: utf-8 -*-from datetime import datetimeimport timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MySlave():    def __init__(self):        pass    def start(self):        for i in range(1, 4):            MyJobWorkerThread(CHANNEL_DISPATCH + '_' + str(i)).start()class MyJobWorkerThread(threading.Thread):    def __init__(self, channel):        threading.Thread.__init__(self)        self.channel = channel    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        p = r.pubsub()        p.subscribe(self.channel)        for message in p.listen():            if message['type'] != 'message':                continue            print("%s: Received dispatched job %s " % (self.channel, message['data']))            print("%s: Run dispatched job %s " % (self.channel, message['data']))            time.sleep(2)            print("%s: Send finished job %s " % (self.channel, message['data']))            ret = r.publish(CHANNEL_RESULT, message['data'])            if ret == 0:                print("%s: Send finished job %s failed." % (self.channel, message['data']))if __name__ == "__main__":    MySlave().start()    time.sleep(10000)

Description

MySlave class-slave node main program, used to start MyJobWorkerThread class-get the distributed job from the channel and send the running result back to the master test. First, run MySlave to define the distribution job channel. Then run MyMaster to dispatch the job and display the execution result.