Python uses Redis for Job Scheduling System (ultra-simple) and redis Scheduling System

Python uses Redis for Job Scheduling System (ultra-simple) and redis Scheduling System

Overview

Redis is an open-source, advanced key-value storage perfect solution for building high-performance, scalable Web applications.

Redis inherits three main features from its many competitions:

The Redis database is completely in the memory, and the disk is used for persistence only.

Compared with many key-value data storage systems, Redis has a rich set of data types.

Redis can copy data to any number of slave servers.

Redis advantages

Exception fast: Redis is very fast. It can execute about 0.11 million sets per second and more than 81000 records per second.

Supports a wide range of data types: Redis supports most developers who already know the image list, set, ordered set, and hash data types. This makes it very easy to solve a variety of problems, because we know which problems can be handled through its data type better.

Operations are atomic: All Redis operations are atomic, which ensures that if the two clients access the Redis server at the same time, the updated value will be obtained.

Multi-function utility: Redis is a multi-utility tool that can be used in multiple applications, such as cache, message, and queue (Redis native supports publishing/subscription), any temporary data, applications, such as Web application sessions and Web page hit count.

Theme entry:

As a typical example of a memory database, Redis has been used in many application scenarios, here, we only talk about how to use the pub/sub function of Redis to implement a simple job scheduling system. I just want to present a simple idea here, so there are still a lot of things to consider that are not included in this example, such as error processing and persistence.

The following are Implementation ideas:

MyMaster: the master node program of the cluster, which is responsible for generating jobs, distributing jobs, and obtaining execution results.

MySlave: The computing node program of the cluster. Each computing node has one. It obtains and runs the job and sends the result to the master node.

Channel CHANNEL_DISPATCH: each slave node subscribes to a channel, for example, "CHANNEL_DISPATCH _ [idx or machine name]". The master will dispatch the publish job to this channel.

Channel CHANNEL_RESULT: the channel used to save the Job results. The master and slave share this channel. The master subscribes to this channel to obtain the job running results. Each slave is responsible for publishing the job execution results to this channel.

Master code

#!/usr/bin/env python# -*- coding: utf-8 -*-import timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MyMaster():def __init__(self):passdef start(self):MyServerResultHandleThread().start()MyServerDispatchThread().start()class MyServerDispatchThread(threading.Thread):def __init__(self):threading.Thread.__init__(self)def run(self):r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)for i in range(1, 100):channel = CHANNEL_DISPATCH + '_' + str(random.randint(1, 3))print("Dispatch job %s to %s" % (str(i), channel))ret = r.publish(channel, str(i))if ret == 0:print("Dispatch job %s failed." % str(i))time.sleep(5)class MyServerResultHandleThread(threading.Thread):def __init__(self):threading.Thread.__init__(self)def run(self):r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)p = r.pubsub()p.subscribe(CHANNEL_RESULT)for message in p.listen():if message['type'] != 'message':continueprint("Received finished job %s" % message['data'])if __name__ == "__main__":MyMaster().start()time.sleep(10000)

Description

MyMaster class-master main program, used to start the thread of dispatch and resulthandler

MyServerDispatchThread class-distribute job threads, generate jobs, and distribute jobs to computing nodes

MyServerResultHandleThread class-job running result processing thread, get job results from channel and display

Slave code

#!/usr/bin/env python# -*- coding: utf-8 -*-from datetime import datetimeimport timeimport threadingimport randomimport redisREDIS_HOST = 'localhost'REDIS_PORT = 6379REDIS_DB = 0CHANNEL_DISPATCH = 'CHANNEL_DISPATCH'CHANNEL_RESULT = 'CHANNEL_RESULT'class MySlave():def __init__(self):passdef start(self):for i in range(1, 4):MyJobWorkerThread(CHANNEL_DISPATCH + '_' + str(i)).start()class MyJobWorkerThread(threading.Thread):def __init__(self, channel):threading.Thread.__init__(self)self.channel = channeldef run(self):r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB)p = r.pubsub()p.subscribe(self.channel)for message in p.listen():if message['type'] != 'message':continueprint("%s: Received dispatched job %s " % (self.channel, message['data']))print("%s: Run dispatched job %s " % (self.channel, message['data']))time.sleep(2)print("%s: Send finished job %s " % (self.channel, message['data']))ret = r.publish(CHANNEL_RESULT, message['data'])if ret == 0:print("%s: Send finished job %s failed." % (self.channel, message['data']))if __name__ == "__main__":MySlave().start()time.sleep(10000)

Description

MySlave class-main program of the slave node, used to start the thread of MyJobWorkerThread

MyJobWorkerThread class-obtain the distributed job from the channel and send the running result back to the master

Test

First, run MySlave to define the dispatch job channel.

Then run MyMaster to dispatch the job and display the execution result.

For Python to use Redis to implement the Job Scheduling System (ultra-simple), I would like to introduce so much to you and hope to help you!

Articles you may be interested in:

• The Laravel framework of PHP is combined with the use and deployment of MySQL and Redis databases.
• Php session Processing Method Based on redis
• Performance of redis's hGetAll function (record Redis's HGETALL)
• Install and deploy Redis in Linux
• Java connection to redis in Vmware
• Configure the srcache_nginx module in Nginx and use Redis to establish a cache system
• How to install and configure a Redis database in CenOS
• Encapsulation Scheme of the mset binary data interface used by C ++ to access Redis
• Optimization of the Redis data import tool developed by C ++
• Talking about the coordinated application of Redis in Distributed Systems