Implementing Message Queuing with Redis

Why Message Queuing is required

The introduction of Message Queuing mechanism in the system is a very big improvement to the system. For example, in a Web system, a user does an action and needs to send an email notification to the user's mailbox. You can use synchronization to allow users to wait for the message to be sent back to the user, but this may cause the user experience to be affected by the network's uncertainty that can result in long waits.

There are scenarios where it is impossible to wait for a synchronization to complete, and those that require a lot of time in the background. For example extreme examples, an on-line compilation system task, 30 minutes to complete the background compilation. The design of this scenario cannot be synchronized after waiting for the feedback, must be the first to feed back the user then asynchronous processing completed, and then wait for the processing to be completed after the user or not feedback.

In addition, the situation of the application of Message Queuing is that the system processing capacity is limited, the first use of the queue mechanism to temporarily store the task, the system to take turns to deal with the queued tasks. In this way, if the system throughput is insufficient, it can handle the high concurrency task stably.

Message Queuing can be used as a queuing mechanism, as long as the system needs to use the queuing mechanism where the message queue can be used.

How to do Message Queuing using Redis

First Redis is designed for caching, but because of its own nature, it can be used to do Message Queuing. It has several blocking APIs to use, and it is these blocking APIs that give him the ability to do Message Queuing.

Redis can do Message Queuing thanks to his list object Blpop Brpop interface and some interfaces of pub/sub (Publish/subscribe). They are all blocking versions, so they can be used to do Message Queuing.

Redis implements FIFO queue

The Redis implementation FIFO is easy and requires only a list object to fetch data from scratch, which can be done from the tail plug data. For example Lpush data, Brpop fetch data.

Redis Implementation Priority queue

First, Brpop and Blpop are supported by multiple list reads, such as Brpop lista listb 0 command can be implemented to read data from the Lista, read the Lista data before reading LISTB data.

Then we can do it in the following ways:

127.0.0.1:6379> lpush a 1 (integer) 1127.0.0.1:6379> lpush a 2 (integer) 2127.0.0.1:6379> lpush a 3 (integer) 3127.0 .0.1:6379> lpush B 1 (integer) 1127.0.0.1:6379> lpush B 2 (integer) 2127.0.0.1:6379> lpush B 3 (integer) 3

127.0.0.1:6379> Brpop a B) "a" 2) "1" 127.0.0.1:6379> Brpop a B) "a" 2) "2" 127.0.0.1:6379> Brpop a B) "a" 2) "3" 127.0.0.1:6379> Brpop a B) "B" 2) "1" 127.0.0.1:6379> Brpop a B) "B" 2) "2" 127.0.0.1:6379> Brpop a B 01) " B "2)" 3 "127.0.0.1:6379> Brpop a B 0

This scenario allows us to support different stages of priority queuing, such as high school low three levels or more.

Multi-priority Problem solving

If there is a large number of priority levels, let's say that there is a requirement that the priority level is not simple high school low or 0-10 of these fixed levels. But it's like 0-99999 of so many levels. Then our third option would be less appropriate.

Although Redis has a sort of data type such as sorted set, it is a pity that it does not have a blocked version of the interface. So we can only use the list type to accomplish the purpose by other means.

There's a simple way we can just set up a queue and make sure it's sorted by priority. Then find a suitable position of the task by the binary search method and insert it to the corresponding position by the LSet command.

For example, the task of writing priority in the queue of bread [1, 3, 6, 8, 9, 14], when there is a priority of 7 of the task comes, we through our own binary algorithm from the queue to take data out of the counter and target data comparison, calculated the corresponding position and then inserted into the designated location.

Because binary lookups are faster, and Redis itself is in memory, the theoretical speed can be guaranteed. But if the amount of data is really large, we can also tune it in some way.

Combining the above scheme will greatly reduce the cost. For example, the queue of data volume 100,000, their priority is also a random 0-100,000 interval. We can set 10 or 100 different queues, 0-10,000 priority tasks to queue 1th, and 10,000-20,000 tasks to queue 2nd. This allows a queue to be split at different levels, and the data for its individual queues is reduced by a number of points, so that the efficiency of the binary lookup match is higher. But the amount of resources that the data occupies is basically the same, and how much memory or how much of the 100,000 data it takes. Just a few more queues in the system.

Redis implements timed Message Queuing

Because the Redis sorting collection (Sorted sets) does not implement blocking, it can only be implemented by the program itself. The score field is stored in a timestamp, because the timestamp is longer we replace it with three digits.

127.0.0.1:6379> zadd seta A (integer) 1127.0.0.1:6379> zadd seta $ b (integer) 1127.0.0.1:6379> zadd seta 300 C (integer) 1127.0.0.1:6379> zadd seta D (integer) 1

First we insert 4 data.

Then we get the data from 0 to the current time. For example, the current timestamp is 200, then we execute the following command

127.0.0.1:6379> zrangebyscore Seta 0 limit 0 each) "a" 127.0.0.1:6379> Zrem seta A (integer) 1127.0.0.1:6379> Zra Ngebyscore SETA 0 201 Limit 0 each) "B" 127.0.0.1:6379> Zrem seta B (integer) 1127.0.0.1:6379> zrangebyscore seta 0 202 Limit 0 1 (empty list or set)

If you take empty data, block a period of time, and then continue to fetch data, loop execution.

Why do we not use the Zremrangebyscore command instead of the Zrangebyscore and Zrem combinations, because Zremrangebyscore does not have the limit parameter and may fetch more than one row of data (e.g. two data socore, etc.), Because concurrency problems can cause Zrem to return 0, it's OK, we continue to fetch.

Java Code snippet:

Public string getdata ()  throws exception {    jedis jedis  = getresource ();    while  (true)  {         set<string> seta = jedis.zrangebyscore ("Seta", 0,  System.currenttimemillis (),  0, 1);        if  (seta  != null && seta.size ()  > 0)  {             string data = seta.toarray (new String[] {}) [0];            long res =  Jedis.zrem ("Seta",  data);            if  (res > 0)  {                 return data;            }        }         thread.sleep (1000L);     }}

From a personal blog: http://www.jflyfox.com/mtg/front/article/997.html

Implementing Message Queuing with Redis