Python implementation of job scheduling system using Redis (Super Simple)

Overview

Redis is an open source, advanced key-value Storage and a perfect solution for building high-performance, scalable Web applications.

The three main features that Redis inherits from its many competitions are:

The Redis database is completely in memory and uses disk for persistence only.

Redis has a rich set of data types compared to many key-value data stores.

Redis can replicate data to any number of slave servers.

Redis Benefits

Exceptionally fast: Redis is very fast and can perform about 110,000 episodes per second, about 81000 + records per second.

Support for rich data types: Redis support Most developers already know like lists, collections, ordered collections, hash data types. This makes it very easy to solve a wide variety of problems because we know which issues are better than the data types that can be handled through it.

Operations are atomic: All Redis operations are atomic, which ensures that Redis servers accessed by two clients at the same time will get the updated values.

Multifunction utility: Redis is a versatile tool that can be used in multiple uses such as cache, message, queue (Redis native support publish/subscribe), any transient data, applications such as Web application sessions, Web page hits count etc.

Step into the topic:

As a typical representation of the in-memory database, Redis has been used in a number of scenarios, where the pub/sub function of Redis is just how to implement a simple job scheduling system. This is just to show a simple idea, so there are still a lot of things to consider that are not included in this example, such as error handling, persistence, and so on.

Here is the idea of implementation

MyMaster: The master node program of the cluster, responsible for generating jobs, distributing jobs, and getting execution results.

Myslave: The compute node program for the cluster, one for each compute node, responsible for getting the job and running it, and sending the results to the master node.

Channel Channel_dispatch: Each slave node subscribes to a CHANNEL, such as "Channel_dispatch_[idx or machine name", Master will publish to this channel the dispatch of the job.

Channel Channel_result: Channel,master and slave that are used to save the results of the job share this channel,master subscribe to this CHANNEL to get job run results. Each slave is responsible for publishing the results of the job execution to this channel.

Master Code

#!/usr/bin/env python#-*-coding:utf-8-*-import timeimport threadingimport randomimport redisREDIS_HOST = ' localhost ' R Edis_port = 6379redis_db = 0channel_dispatch = ' Channel_dispatch ' channel_result = ' Channel_result ' class MyMaster ():d EF _ _init__ (self):p assdef start: Myserverresulthandlethread (). Start () Myserverdispatchthread (). Start () class Myserverdispatchthread (Threading. Thread):d EF __init__ (self): threading. Thread.__init__ (self) def run (self): R = Redis. Strictredis (Host=redis_host, Port=redis_port, db=redis_db) for I in range (1, +): Channel = Channel_dispatch + ' _ ' + str (RA Ndom.randint (1, 3)) print ("Dispatch job%s to%s"% (str (i), channel)) ret = R.publish (channel, str (i)) if ret = = 0:print ("Di Spatch job%s failed. "% str (i)) Time.sleep (5) class Myserverresulthandlethread (threading. Thread):d EF __init__ (self): threading. Thread.__init__ (self) def run (self): R = Redis. Strictredis (Host=redis_host, Port=redis_port, db=redis_db) p = r.pubsub () p.subscribe (Channel_result) for message in P.listen(): If message[' type ']! = ' message ': Continueprint ("Received finished job%s"% message[' data ']) if __name__ = = "__main__": M Ymaster (). Start () Time.sleep (10000)

Description

MyMaster Class-Master main program, the thread used to start dispatch and Resulthandler

Myserverdispatchthread class-distributes job threads, generates jobs, and distributes them to compute nodes

Myserverresulthandlethread class-Job run result processing thread, obtaining job results from channel and displaying

Slave code

#!/usr/bin/env python#-*-coding:utf-8-*-from datetime import datetimeimport timeimport threadingimport Randomimport re Disredis_host = ' localhost ' redis_port = 6379redis_db = 0channel_dispatch = ' Channel_dispatch ' Channel_result = ' CHANNEL_ RESULT ' class Myslave ():d ef __init__ (self):p assdef start: For I in range (1, 4): Myjobworkerthread (Channel_dispatch + ' _ ' + str (i)). Start () class Myjobworkerthread (threading. Thread):d EF __init__ (self, Channel): Threading. Thread.__init__ (self) Self.channel = Channeldef Run (self): R = Redis. Strictredis (Host=redis_host, Port=redis_port, db=redis_db) p = r.pubsub () p.subscribe (Self.channel) for message in P.listen (): If message[' type ']! = ' message ': Continueprint ("%s:received dispatched job%s"% (Self.channel, message[' data ']) print ("%s:run dispatched job%s"% (Self.channel, message[' data ')) Time.sleep (2) print ("%s:send finished job%s"% (Self.channel, message[' data ')) ret = R.publish (channel_result, message[' data ']) if ret = = 0:print ("%s:send finished Job%s failed. "% (Self.channel, message[' data"))) if __name__ = = "__main__": Myslave (). Start () Time.sleep (10000) 

Description

Myslave class-Slave node main program, the thread used to start Myjobworkerthread

Myjobworkerthread class-Get the dispatched jobs from the channel and send the results back to master

Test

Run Myslave first to define the dispatch channel.

Then run the MyMaster dispatch job and display the results of the execution.

About Python implementation of the job scheduling system using Redis (Super Simple), small series on the introduction of so many, I hope to help you!

