How to implement Redis queue priority code examples

How to do Message Queuing using Redis

First Redis is designed for caching, but because of its own nature, it can be used to do Message Queuing. It has several blocking APIs to use, and it is these blocking APIs that give him the ability to do Message Queuing.

Imagine that you can do this without using Message Queuing in the "database solve all problems" mentality. We keep all the tasks in the database and then take the task processing through continuous polling. This approach can do your job, but it's a poor practice. But if your database interface provides a blocking method then you can avoid polling, and your database can be used to do Message Queuing, but the current database does not have such an interface.

Other features of Message Queuing, such as FIFO, are also easy to implement, requiring only a list object to fetch data from scratch, which can be implemented from the tail plug data.

Redis can do Message Queuing thanks to his list object Blpop Brpop interface and some interfaces of pub/sub (Publish/subscribe). They are all blocking versions, so they can be used to do Message Queuing.

Priority practices for RABBITMQ

There are many well-known Message Queuing products in the current mature, such as RABBITMQ. It is relatively simple to use, the function is relatively rich, the general situation is fully sufficient. But one thing that's annoying is that it doesn't support precedence.

For example, a mail-sending task, some privileged users want its mail to be sent out more timely, at least more than the average user priority. By default RABBITMQ cannot be disposed of, and the task thrown to RABBITMQ is FIFO first-out. However, we can use some flexible techniques to support these priorities. Create multiple queues and set up the appropriate routing rules for RABBITMQ consumers.

For example, by default there is such a queue, we take a list to simulate [Task1, Task2, Task3], the consumer takes turns according to FIFO principle each take out task to dispose of. If a high-priority task comes in, it can only be followed at the end of the process [Task1, Task2, Task3, Higitask1]. But if you use two queues, a high-priority queue, and a normal-priority queue. Normal priority [Task1, Task2, TASK3], high priority [HIGHTASK1] then we set the consumer's route to let the consumer randomly fetch data from any queue.

And we can define a consumer who specializes in high-priority queues, and it doesn't handle low-priority queue data when it's idle. This is similar to the VIP counter of the bank, ordinary customers in the bank to queue up, a VIP came he although not from the pick-up machine in front of the ordinary members of the ticket, but he still can go straight to the VIP channel faster.

Use RABBITMQ to do a priority message queue, as described above with the bank VIP members, take different channels. But this approach is only a relative priority and does not have absolute priority control, for example, I want a high priority task to be in absolute priority than other common tasks, so the above scenario is not feasible. Because RABBITMQ consumers only know that their own idle situation from the queue of their own "random" to take a queue of the first data to deal with, it can not control the priority to find which queue. or finer-grained priority control. Or there are more than 10 priority settings in your system. This is also difficult to achieve with RABBITMQ.

But if you use Redis to do the queue, the requirements above can be fulfilled.

Why Message Queuing is required

The introduction of Message Queuing mechanism in the system is a very big improvement to the system. For example, in a Web system, a user does an action and needs to send an email notification to the user's mailbox. You can use synchronization to allow users to wait for the message to be sent back to the user, but this may cause the user experience to be affected by the network's uncertainty that can result in long waits.

There are scenarios where it is impossible to wait for a synchronization to complete, and those that require a lot of time in the background. For example extreme examples, an on-line compilation system task, 30 minutes to complete the background compilation. The design of this scenario cannot be synchronized after waiting for the feedback, must be the first to feed back the user then asynchronous processing completed, and then wait for the processing to be completed after the user or not feedback.

In addition, the situation of the application of Message Queuing is that the system processing capacity is limited, the first use of the queue mechanism to temporarily store the task, the system to take turns to deal with the queued tasks. In this way, if the system throughput is insufficient, it can handle the high concurrency task stably.

Message Queuing can be used as a queuing mechanism, as long as the system needs to use the queuing mechanism where the message queue can be used.

Implementation of Redis Message queue priority

A description of some basic Redis basics

Redis> blpop tasklist 0 "im task 01"

This example uses the Blpop command to block the way a data is taken from the Tasklist list, and the last parameter is the time to wait for a timeout. If set to 0, it indicates an infinite wait. In addition, Redis can only store data in string type, so it can only pass strings when the task is passed. We simply have to serialize the responsible data into a JSON-formatted string, and then the consumer can convert it.

Here our sample language uses Python, and the libraries that link Redis use Redis-py. If you have some programming basics to switch it into your favorite language it should be okay.

1. Simple FIFO queue

Import Redis, Timedef handle (Task):    Print Task    time.sleep (4) def main ():    pool = Redis. ConnectionPool (host= ' localhost ', port=6379, db=0)    r = Redis. Redis (Connection_pool=pool) while    1:        result = R.brpop (' tasklist ', 0)        handle (result[1]) if name = = "Main": C7/>main ()

On the example, even for the simplest consumer, we are constantly fetching data from the Redis queue through an infinite loop. If there is no data in the queue and there is no time-out, the data is removed and executed down.

The general case is a complex string, we may need to format it as a re-pass to the handler function, but for the sake of simplicity our example is a normal string. Another example of the processing function does not do any processing, just sleep to simulate time-consuming operations.

We also open a Redis client to simulate the producer, and the client that comes with it can. Go to the tasklist queue and plug in some data.

redis> lpush tasklist ' im task ' redis> lpush tasklist ' im task ' redis> lpush tasklist ' im task ' redis> l Push tasklist ' im task redis> lpush tasklist ' im task 05 '

The simulated tasks are then consumed at the consumer's end.

2. Simple-priority Queue

Assuming a simple requirement, only high-priority tasks that are higher than the low-priority task are first disposed of. The order of the other tasks is not the same, we just need to push it to the front of the queue when we encounter a high-priority task, rather than pushing to the last.

Because our queue is a list of Redis that is used, it is easy to implement. Encountering high-priority usage Rpush encounters a low-priority use Lpush

redis> lpush tasklist ' im task ' redis> lpush tasklist ' im task ' redis> rpush tasklist ' im high task ' redis& Gt Rpush tasklist ' im high task "redis> lpush tasklist ' im task ' redis> rpush tasklist ' im high task 03 '

You will then see that high priority is always the first to execute with a lower priority. But the disadvantage of this scenario is that the order of execution between high-priority tasks is advanced.

3. A more complete queue

Example 2 simply plugs a high-priority task into the front of the queue, and the lower-priority is plugged into the last side. This does not guarantee the order between high-priority tasks.

Assuming that all tasks are high-priority, the order in which they are executed will be reversed. This clearly violates the FIFO principle of the queue.

But just a little improvement can improve our queue.

As with RABBITMQ, we set up two queues, one with a high priority and one low priority. High-priority tasks are placed in high queues and low in low-priority queues. The difference between Redis and RABBITMQ is that it can require queue consumers to read from which queue.

def main ():    pool = Redis. ConnectionPool (host= ' localhost ', port=6379, db=0)    r = Redis. Redis (Connection_pool=pool)    while 1:        result = R.brpop ([' High_task_queue ', ' low_task_queue '], 0)        handle ( RESULT[1])

The above code will block the data from the two queues of ' high_task_queue ', ' low_task_queue ', if the first one is not taken from the second one.

So only the queue of consumers to do such an improvement can achieve the goal.

redis> lpush low_task_queue low001redis> lpush low_task_queue low002redis> lpush low_task_queue Low003redis > Lpush low_task_queue low004redis> lpush high_task_queue low001redis> lpush high_task_queue low002redis> Lpush high_task_queue low003redis> Lpush high_task_queue low004

The above tests show that the high priority is the first to be executed, and that the FIFO principle is ensured between the high priority levels.

This scenario allows us to support different stages of priority queuing, such as high school low three levels or more.

4. A lot of priority level situations

Suppose there is such a requirement that the priority level is not simple high school low or 0-10 of these fixed levels. But it's like 0-99999 of so many levels. Then our third option would be less appropriate.

Although Redis has a sort of data type such as sorted set, it is a pity that it does not have a blocked version of the interface. So we can only use the list type to accomplish the purpose by other means.

There's a simple way we can just set up a queue and make sure it's sorted by priority. Then find a suitable position of the task by the binary search method and insert it to the corresponding position by the LSet command.

For example, the task of writing priority in the queue of bread [1, 3, 6, 8, 9, 14], when there is a priority of 7 of the task comes, we through our own binary algorithm from the queue to take data out of the counter and target data comparison, calculated the corresponding position and then inserted into the designated location.

Because binary lookups are faster, and Redis itself is in memory, the theoretical speed can be guaranteed. But if the amount of data is really large, we can also tune it in some way.

Thinking back to our third scenario, the combination of a third scenario would greatly reduce the overhead. For example, the queue of data volume 100,000, their priority is also a random 0-100,000 interval. We can set 10 or 100 different queues, 0-10,000 priority tasks to queue 1th, and 10,000-20,000 tasks to queue 2nd. This allows a queue to be split at different levels, and the data for its individual queues is reduced by a number of points, so that the efficiency of the binary lookup match is higher. But the amount of resources that the data occupies is basically the same, and how much memory or how much of the 100,000 data it takes. Just a few more queues in the system.