Using Redis to implement Message Queuing with priority

Objective

Previously there has been the use of celery priority mechanism (Redis-based task queue), has been very curious about its implementation mechanism, after consulting some of the information, decided to write this article, as a summary.

1. Using sorted Set implementation

The greatest advantage of using sorted Set for priority queuing is its intuitive clarity.

...]

Score as the priority, member as the corresponding task
In sorted Set, the score is small, in the head of the priority queue, which is higher priority
Because score is the priority of menber, it is very intuitive
can use

key0000EXEC

To get the highest-priority element in the task queue
Zrange used to get tasks, Zremrangebyrank to remove from message queue

Note : Because the sorted set itself is a set, messages in Message Queuing cannot be duplicated, otherwise the newly added message overwrites the previously joined message
Note : For score the same message, Sorted Set is sorted by dictionary order

2. Using list implementation

It should be possible to think of a list as the best choice for Message Queuing, but there are several different ways to implement a prioritized message queue using list.

2.1 Preparation

First, if we assume that messages in Message Queuing are pushed from the right side of the message queue (Rpush), take out from the left (Lpop)
Then a single list can easily be constructed into a FIFO queue. But if the priority level is only two, high and low, then we can push high priority messages, use Lpush to the left side of the queue, the low priority messages, use Rpush push to the right side of the queue, so that a single list can achieve a 2-level priority message queue.

2.2 Using Blpop

Redis provides a list of blocked (blocking) pop-up primitives.

...] timeout

When a number of key parameters are given, the first element of a non-empty list is ejected by checking each list in the order of key parameters.
So we can create three queues, High,normal, low, representing high priority, normal priority, lower priority, respectively.

normal low

2.3 Lpop based on multiple keys

Sometimes we do not want to block the primitive, then in the business layer, we can traverse in multiple queues, find to get the message

queue_list = ["high""normal""low"]def get_msg():    forin queue_list：        msg = redis_db.lpop(queue)        ifisnotNone:            return msg    returnNone

When I looked through the source of RQ, I found that the task queue with the priority of RQ was implemented in this way.

2.4 Extensions

If we need 10 priority message queues, we can expect that we need at least 5 queues (refer to 2.1)
The naming of our message queue may require some sort of rule
For example, the name of the message queue originally intended to be named Msg_queue
Then these 5 message queues can be named
msg_queue-0
msg_queue-1
msg_queue-2
msg_queue-3
Msg_queue-4

If we combine

KEYS pattern

We can get a message queue that supports any number of priority levels

# priority 1 ~ Ten# Push message into list def push_message(queue, priority, message):num = (Priority-1) /2Target_queue = queue +"-"+ str (num)# Direct    ifPriority%2==1: Redis_db.lpush (target_queue, message)Else: Redis_db.rpush (target_queue, message)# Fetch a message def fetch_message(queue):Queue_list = Redis_db.keys (queue +"-?") Queue_list = sorted (queue_list) forQueueinchqueue_list:msg = Redis_db.lpop (queue)ifMsg is  not None:returnMsgreturn None

Note : With this approach, messages of the same priority do not meet the FIFO

Using Redis to implement Message Queuing with priority