Python uses Redis to implement a simple job scheduling system

Source: Internet
Author: User

Python uses Redis to implement a simple job scheduling system
Python uses Redis to implement a simple job scheduling system

As a typical example of a memory database, Redis has been used in many application scenarios, here, we only talk about how to use the pub/sub function of Redis to implement a simple job scheduling system. I just want to present a simple idea here, so there are still a lot of things to consider that are not included in this example, such as error processing and persistence.

The following are Implementation ideas:

MyMaster: the master node program of the cluster, which is responsible for generating jobs, distributing jobs, and obtaining execution results. MySlave: The computing node program of the cluster. Each computing node has one. It obtains and runs the job and sends the result to the master node. Channel CHANNEL_DISPATCH: each slave node subscribes to a channel, for example, "CHANNEL_DISPATCH _ [idx or machine name]". The master will dispatch the publish job to this channel. Channel CHANNEL_RESULT: the channel used to save the Job results. The master and slave share this channel. The master subscribes to this channel to obtain the job running results. Each slave is responsible for publishing the job execution results to this channel. Master code
#!/usr/bin/env python# -*- coding: utf-8 -*-import timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MyMaster():    def __init__(self):        pass    def start(self):        MyServerResultHandleThread().start()        MyServerDispatchThread().start()class MyServerDispatchThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        for i in range(1, 100):            channel = CHANNEL_DISPATCH + '_' + str(random.randint(1, 3))            print("Dispatch job %s to %s" % (str(i), channel))            ret = r.publish(channel, str(i))            if ret == 0:                print("Dispatch job %s failed." % str(i))            time.sleep(5)class MyServerResultHandleThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        p = r.pubsub()        p.subscribe(CHANNEL_RESULT)        for message in p.listen():            if message['type'] != 'message':                continue            print("Received finished job %s" % message['data'])if __name__ == "__main__":    MyMaster().start()    time.sleep(10000)

Description

MyMaster class-master main program, used to start dispatch and resulthandler threads MyServerDispatchThread class-dispatch job threads, generate jobs and distribute them to the compute node MyServerResultHandleThread class-job running result processing thread, obtain the job result from the channel and display the Slave code.
#!/usr/bin/env python# -*- coding: utf-8 -*-from datetime import datetimeimport timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MySlave():    def __init__(self):        pass    def start(self):        for i in range(1, 4):            MyJobWorkerThread(CHANNEL_DISPATCH + '_' + str(i)).start()class MyJobWorkerThread(threading.Thread):    def __init__(self, channel):        threading.Thread.__init__(self)        self.channel = channel    def run(self):        r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)        p = r.pubsub()        p.subscribe(self.channel)        for message in p.listen():            if message['type'] != 'message':                continue            print("%s: Received dispatched job %s " % (self.channel, message['data']))            print("%s: Run dispatched job %s " % (self.channel, message['data']))            time.sleep(2)            print("%s: Send finished job %s " % (self.channel, message['data']))            ret = r.publish(CHANNEL_RESULT, message['data'])            if ret == 0:                print("%s: Send finished job %s failed." % (self.channel, message['data']))if __name__ == "__main__":    MySlave().start()    time.sleep(10000)

Description

MySlave class-slave node main program, used to start MyJobWorkerThread class-get the distributed job from the channel and send the running result back to the master test. First, run MySlave to define the distribution job channel. Then run MyMaster to dispatch the job and display the execution result.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.