Redis Implementation Queue

Reference: "Redis Getting Started Guide" 4th Chapter Advanced

Http://book.51cto.com/art/201305/395461.htm

4.4.2 using Redis to implement task queues

When it comes to queues, it's natural to think of Redis's list types, 3.4.2 describes the concept of using the Lpush and Rpop commands to implement queues. If you want to implement a task queue, you only need to have the producer use the Lpush command to join a key, and let the consumer use the Rpop command to remove the task from the key in a constant way.

In the small white example, the task of completing the email needs to know the address, message subject, and message body. So the producer needs to make these three messages into an object and serialize them into a string, and then add them to the task queue. The consumer then loops through the queue to pull the task, like this pseudo-code:

1. # infinite Loop reads the contents of the task queue
2. Loop
3. $task = rpor Queue
4. If $task
5. # If there is a task in the task queue then execute it
6. Execute ($task)
7. Else
8. # wait 1 seconds to avoid requesting data too frequently if not
9. Wait 1 second

To this a simple task queue that uses Redis implementation is ready to be written. But there is a bit of imperfection: when there are no tasks in the task queue, the consumer calls the Rpop command once per second to see if there are any new tasks. It would be nice to notify the consumer once a new task is added to the task queue. In fact, with the help of brpop command can achieve such a demand.

The Brpop command is similar to the Rpop command, except that when there are no elements in the list, the Brpop command blocks the connection until a new element is added. As the previous segment of the code can be rewritten as:

1. Loop
2. # If there are no new tasks in the task queue, the Brpop command will block until execute () is executed.
3. $task = brpop queue, 0
4. # The return value is an array (see below), the second element of the array is the task we need.
5. Execute ($task [1])

The Brpop command receives two parameters, the first is the key name, the second is the time-out, and the unit is seconds. Nil is returned when the new element is still not obtained at this time. The timeout in the previous example is "0", which means that there is no limit on the time to wait, that is, if no new element is added to the list, it will be blocked forever.

When an element is obtained , the Brpop command returns two values, namely the key name and the element value. For the test brpop command, we can open two REDIS-CLI instances in instance a:

1. Redis A> Brpop Queue 0

Brpop:

return value:

If no element is ejected within the specified time, aNilAnd wait for the length of time. Instead, a list of two elements is returned, and the first element is the one to which the popup element belongs.Key, the second element is the value of the element being popped.

When you type carriage return, instance 1 is blocked, and an element is added to the queue in instance B:

1. Redis B> Lpush Queue Task
2. (integer) 1

After the Lpush command executes, instance a immediately returns the result:

1. 1) "Queue"
2. 2) "Task"

You will also find that the elements in the queue have been taken away:

1. Redis> Llen queue
2. (integer) 0

In addition to the Brpop command, Redis also provides Blpop, and the difference between Brpop and Blpop is taken from the left side of the queue when the element is taken from the queue. Concrete can refer to lpop understanding, here no longer repeat.

4.4.3 Priority Queue

It's a little white blog. You need to send a message to each Subscriber when the article is published, and this step can also be accomplished using the task queue. Because the tasks to be performed are the same as sending a confirmation message, they can share a single consumer. However, imagine a scenario where a user who subscribes to a small white blog has 1000 people, and when a new article is published, the blog adds 1000 tasks to send notification messages to the task queue. If it takes 10 seconds for each message to be sent, it will take nearly 3 hours to complete all 1000 tasks. The problem is, if there are new users who want to subscribe to the small white blog, when he submitted his mailbox and saw the Web page prompting him to check the confirmation email, he did not know that the task of sending himself a confirmation email was added to the queue already has 1000 tasks. To receive a confirmation email, he had to wait for nearly 3 hours. What a bad user experience! On the other hand, it is not urgent to notify subscribers when a new article is posted, and most users do not require a new article to be notified immediately, even a delay of one day can be acceptable in many cases.

So it is possible to conclude that the first two tasks of sending a confirmation email and sending a notification message exist simultaneously. To achieve this, we need to implement a priority queue.

The Brpop command can receive multiple keys at the same time, with the full command format Brpop key [key ...] timeout,

check each list in the order of key parameters and pop up the trailing element of the first non-empty list

such as Brpop queue:1 Queue:2 0. The meaning is to detect multiple keys at the same time, if none of the keys are blocked, and if one of the keys has an element, the element is ejected from the key . For example, open two REDIS-CLI instances, in instance a:

1. Redis A> Brpop queue:1 queue:2 queue:3 0

In instance B:

1. Redis B> Lpush queue:2 Task
2. (integer) 1

The instance a will return:

1. 1) "Queue:2"
2. 2) "Task"

If multiple keys have elements, an element in the first key is taken from left to right. We first add an element to Queue:2 and Queue:3:

1. Redis> Lpush queue:2 task1
2. 1) (integer) 1
3. Redis> Lpush queue:3 task2
4. 2) (integer) 1

Then execute the brpop command:

1. Redis> Brpop queue:1 queue:2 queue:3 0
2. 1) "Queue:2"
3. 2) "Task1"

This attribute enables priority-sensitive task queues to be implemented. We use queue:confirmation respectively. Email and queue:notification.email Two keys store two tasks, sending a confirmation email and sending a notification message, and then changing the consumer's code to:

1. Loop
2. $task =
3. Brpop Queue:confirmation.email,
4. Queue:notification.email,
5. 0
6. Execute ($task [1])

Once the task to send the confirmation message is added to the Queue:confirmation.email queue, regardless of how many tasks the Queue:notification.email has, the consumer will prioritize the task of sending a confirmation email.

Reference: http://book.51cto.com/art/201305/395463.htm

Website statement:

Rpoplpush Source Destination

Command Rpoplpush performs the following two actions within an atomic time:

• POPs the last element in the list source (the trailing element) and returns it to the client.
• Inserts a source popup element into the list destination , as the head element of the destination list.

Mode: Secure queue

The list of Redis is often used as a queue to exchange messages (message) in an orderly manner between different programs. One client puts the message into the queue through the Lpush command, while another client takes the longest waiting message in the queue with the Rpop or Brpop command.

Unfortunately, the queue method above is "unsafe" because in this process a client may crash after extracting a message, and the messages that are not processed are lost.

Using the Rpoplpush command (or its blocking version Brpoplpush) solves this problem: because it not only returns a message, it also adds the message to another backup list, and if everything works, when a client finishes processing a message, it can use the Lrem command to remove this message from the backup table.

Finally, you can add a client specifically for monitoring the backup table, which automatically puts the message over a certain processing time back into the queue (the client responsible for processing the message may have crashed) so that no messages are lost.

Mode: Circular List of circular lists

By using the same key as the two parameters of the Rpoplpush command, the client can get all the elements of the list in a way that gets the list elements one after the other, without having to transfer all the list elements from the server to the client as quickly as the Lrange command does ( The total complexity of both methods is O (N).

The above pattern works even in the following two cases:

• Multiple clients rotate the same list at the same time (rotating), and they get different elements until all the elements have been read and then start from scratch.
• A client adds a new element to the tail of the list (to the right).

This pattern makes it easy to implement a system that has N clients that need to process some elements continuously, and process them as fast as possible. A typical example is the server's monitoring program: they need to check a set of Web sites in parallel, in the shortest possible time, to ensure their accessibility.

Note that the client using this mode is easy to extend (Scala) and secure (reliable), because even if the client that receives the element fails, the element remains in the list, is not lost, and when the next iteration arrives, other clients can continue to process the elements.

More:

Http://redis.io/commands/blpop

\http://redis.io/commands/rpoplpush

Redis Implementation Queue