Use redis to Implement Message queues that support priority

Use redis to Implement Message queues that support priority

 

Why is the introduction of the message queue mechanism in the message queue system greatly improving the system. For example, in a web system, a user needs to send an email notification to the user's mailbox after performing an operation. You can use the synchronization method to allow users to wait for the mail to be sent to the user, but this may affect the user experience due to the uncertainty of the network. In some scenarios, the synchronization method cannot be used to wait for completion. operations that require a large amount of time in the background are required. For example, in an extreme example, it takes 30 minutes for an online compilation system task to complete background compilation. The design of this scenario is impossible to give feedback after waiting for synchronization. It is necessary to first give feedback to the user and then complete asynchronous processing, and then wait for the feedback to be sent to the user based on the situation. In addition, message queues are applicable to scenarios where the processing capability of the system is limited. The queue mechanism is used to temporarily store tasks, and the system takes turns to process queued tasks one by one. In this way, high concurrency tasks can be processed stably when the system throughput is insufficient. Message Queue can be used for the queuing mechanism. As long as the system needs to use the queuing mechanism, Message Queue can be used. Rabbitmq has many mature MQ products, such as rabbitmq. It is relatively simple to use and has rich functions. Generally, it is enough. However, it does not support priority. For example, for an email task, some privileged users want their emails to be sent in a more timely manner, which is at least preferred for normal users. By default, rabbitmq cannot be processed. all tasks that are thrown to rabbitmq are FIFO. However, we can use some workarounds to support these priorities. Create multiple queues and set corresponding routing rules for rabbitmq consumers. For example, there is such a queue by default. We use the list to simulate [task1, task2, task3]. The consumer takes turns to take tasks one by one based on the FIFO principle. If a task with a higher priority comes in, it can only be processed at the end of [task1, task2, task3, higitask1]. However, if two queues are used, one is a high-priority queue, and the other is a normal priority queue. The general priority is [task1, task2, task3], and the high priority is [hightask1]. Then, we set the consumer route so that the consumer can randomly retrieve data from any queue. In addition, we can define a consumer that specifically processes high-priority queues. When it is idle, it does not process data in low-priority queues. Similar to the VIP counter of a bank, a common customer queues for receiving numbers from a bank. Although a VIP arrives, he does not take out a ticket from the number receiving machine that is in front of a common member, but he can still go through the VIP channel faster. If rabbitmq is used as a message queue that supports priority, it uses different channels like the VIP members in the bank described above. However, this method only has a relative priority and cannot be controlled by absolute priority. For example, I want a task with a higher priority to take precedence over other common tasks, in this way, the above solution does not work. Because rabbitmq consumers only get the first data in a queue randomly when they are idle, it cannot control which queue to first find. Or more fine-grained priority control. Or you can set more than 10 priorities in the system. This is also difficult to implement using rabbitmq. However, if you use redis for the queue, the above requirements can be met. How to Use redis for message queue first, redis is designed for caching, but it can be used for message queue due to its own characteristics. It has several blocking APIs that can be used. These blocking APIs enable him to queue messages. Imagine that, with the idea of "solving all database problems", you can achieve your needs without using message queues. We store all the tasks in the database and process them through continuous polling. This method can accomplish your tasks, but it is very poor. However, if your database interface provides a blocking method, you can avoid polling operations. Your database can also be used for message queues, but the current database does not have such an interface. In addition, other features of message queue, such as FIFO, are also easy to implement. You only need a list object to fetch data from the beginning and plug data from the end. Redis can do message queues thanks to its list object blpop brpop interface and some pub/SUB (publish/subscribe) interfaces. They are all blocking versions, so they can be used for message queues. Implementation of redis message queue priority some basic redis knowledge

redis> blpop tasklist 0"im task 01"

In this example, the blpop command gets the first data from the tasklist in blocking mode. The last parameter is the waiting time. If it is set to 0, it means unlimited waiting. In addition, redis can only store data of the string type. Therefore, only strings can be transferred during Task transmission. We only need to simply serialize the data into a JSON string, and then the consumer can convert it. Here we use python as the example language and redis-py as the link library. If you switch it to your preferred language based on some programming basics, it should be okay. 1. Simple FIFO queue

import redis, time
def handle(task):    print task    time.sleep(4)
def main():    pool = redis.ConnectionPool(host=‘localhost‘, port=6379, db=0)    r = redis.Redis(connection_pool=pool)    while 1:        result = r.brpop(‘tasklist‘, 0)        handle(result[1])
if __name__ == "__main__":    main()

In the above example, even the simplest consumer, we continuously retrieve data from the redis queue through an infinite loop. If there is no data in the queue, there is no timeout blocking, and if there is data, it is taken down and executed. Generally, it is a complex string. We may need to format it and then pass it to the processing function. However, for the sake of simplicity, our example is a common string. In addition, the processing function in the example does not process anything, and only sleep is used to simulate time-consuming operations. We have another redis client to simulate the producer, and the built-in client can be used. Add more data to the tasklist queue.

redis> lpush tasklist ‘im task 01‘redis> lpush tasklist ‘im task 02‘redis> lpush tasklist ‘im task 03‘redis> lpush tasklist ‘im task 04‘redis> lpush tasklist ‘im task 05‘ 

Then, on the consumer side, we can see that these simulated tasks are consumed one by one. 2. A simple priority queue is assumed to be a simple requirement, and only tasks with higher priority and lower priority need to be processed first. The order of other tasks does not matter. In this way, we only need to put it in front of the queue when encountering a high-priority task, rather than pushing it to the end. Because our queue is the list of redis, it is easy to implement. Use rpush with a higher priority and use lpush with a lower priority

redis> lpush tasklist ‘im task 01‘redis> lpush tasklist ‘im task 02‘redis> rpush tasklist ‘im high task 01‘redis> rpush tasklist ‘im high task 01‘redis> lpush tasklist ‘im task 03‘redis> rpush tasklist ‘im high task 03‘

Later, we will see that higher priority is always the first execution than lower priority. However, the disadvantage of this solution is that the execution sequence between high-priority tasks is advanced. 3. In the more perfect queue example 2, only high-priority tasks are put at the top of the queue, and low-priority tasks are put at the end. In this way, the order between high-priority tasks cannot be ensured. Assuming that all tasks have a high priority, their execution order is the opposite. This obviously violates the FIFO principle of the queue. However, our queue can be improved with a little improvement. Like using rabbitmq, we set two queues, one with a high priority and the other with a low priority. High-priority tasks are put in the high queue, and low-priority tasks are put in the low-priority queue. Redis and rabbitmq are different in that they can require the queue consumers to read from first.

def main():    pool = redis.ConnectionPool(host=‘localhost‘, port=6379, db=0)    r = redis.Redis(connection_pool=pool)    while 1:        result = r.brpop([‘high_task_queue‘, ‘low_task_queue‘], 0)        handle(result[1])

The above code gets data from the 'High _ task_queue 'and 'low _ task_queue' queues in a blocking manner. If the first queue is not retrieved from the second queue. Therefore, you only need to improve the queue consumer.

redis> lpush low_task_queue low001redis> lpush low_task_queue low002redis> lpush low_task_queue low003redis> lpush low_task_queue low004redis> lpush high_task_queue low001redis> lpush high_task_queue low002redis> lpush high_task_queue low003redis> lpush high_task_queue low004

From the test above, we can see that high-priority instances are executed first, and the FIFO principle is ensured between high-priority instances. This solution supports priority queues of different stages, such as three levels of high and low levels or more levels. 4. In many cases, the priority level is assumed to have such a requirement. The priority is not a fixed level of simple high school or low level or 0-10. It is similar to 0-99999. The third solution is not suitable. Although redis has sorted set and other data types that can be sorted, it is a pity that it does not block the interface of the version. Therefore, we can only use the list type to achieve the goal through other methods. In a simple way, we can set only one queue and ensure that it is sorted by priority. Then, you can use the binary search method to find a proper job location and insert it to the appropriate location using the lset command. For example, a queue contains a write priority task [1, 3, 6, 8, 9, 14]. When a task with a priority of 7 comes in, we use our own binary algorithm to retrieve data from the queue one by one and compare it with the target data, calculate the corresponding location, and insert it to the specified location. Because binary search is relatively fast, and redis itself is in the memory, the speed can be guaranteed theoretically. However, if the data volume is indeed large, we can optimize it in some ways. Looking back at our third solution, combining the third solution will greatly reduce the overhead. For example, for a queue with a data volume of 100,000, their priority ranges from 0 to 100,000. We can set 10 or 100 different queues, with 0-10 thousand priority tasks serving on queue 1, and 10 thousand-20 thousand tasks serving on queue 2. In this way, after a queue is split by different levels, the data in a single queue will be much less, so that the efficiency of binary search and matching will be higher. However, the resources occupied by data are basically unchanged, and the amount of memory occupied by 100,000 of data remains unchanged. Only some queues are added to the system.