Redis Research (15)-Task queue

In Web site development, page rendering is blocked when the page needs to perform lengthy operations such as sending mail, complex data operations, and more. To prevent users from waiting too long, you should use a separate thread to do this kind of operation. However, some programming languages or frameworks are not easy to implement multi-threading, which can easily be thought of by other processes. Imagine that a process can complete the sending of e-mail, then in the page only need to find a way to notify the process to the specified address to send mail.

The notification process can be accomplished with a task queue . The task queue, as its name implies, is the queue for delivery tasks. There are two types of entities that interact with the task queue, one for the producer (producer) and one for the consumer (consumer). The producer places the tasks that need to be processed into the task queue, and the consumer continuously reads the task information from the task queue and executes it.

For e-Mail This operation, the page program is the producer, and the process of sending mail is the consumer. When a message needs to be sent, the page program assembles the Inbox address, message subject, and message body into a task queue. The process of sending mail will constantly check the task queue, and once a new task is found, it will be removed from the queue and

Yes. Thus, the communication between processes is realized.

using task queues has the following benefits:
(1) Loose coupling. Producers and consumers do not need to know each other's implementation details, only the description format of the task is agreed. This enables producers and consumers to be written by different teams using different programming languages.

(2) Easy to expand consumers can have multiple, and can be distributed in different servers, this can easily reduce the load on a single server.

Below we use Redis to implement the task queue

The list type of Redis, using the Lpush and Rpop commands to implement the queue . If you want to implement a task queue, you only need to have the producer use the Lpush command to join a key, and let the consumer use the Rpop command to remove the task from the key in a constant way.

In the example above, the task of completing the email needs to know the recipient address, the subject of the message, and the message body. So the producer needs to make these three messages into an object and serialize them into a string, and then add them to the task queue. The consumer then loops through the queue to pull the task, like this pseudo-code:

#无限循环读取任务队列中的内容loop $task =rpor Queueif $task # If there is a task in the task queue execute it execute ($task) else# if not, wait 1 seconds to avoid requesting data too frequently wait 1 second

To this a simple task queue that uses Redis implementation is ready to be written. But there is a bit of imperfection: when there are no tasks in the task queue, the consumer calls the Rpop command once per second to see if there are any new tasks. It would be nice to notify the consumer once a new task is added to the task queue. In fact, with the help of brpop command can achieve such a demand.

The Brpop command is similar to the Rpop command, except that when there are no elements in the list, the Brpop command blocks the connection until a new element is added . As the previous segment of the code can be rewritten as:

loop# If there are no new tasks in the task queue, the Brpop command will block until execute () is executed. $task =brpop queue, the 0# return value is an array (see below), and the second element of the array is the task we need. Execute ($task [1])

The Brpop command receives two parameters, the first is the key name, the second is the time-out, and the unit is seconds. Nil is returned when the new element is still not obtained at this time. The timeout in the previous example is "0", which means that there is no limit on the time to wait, that is, if no new element is added to the list, it will be blocked forever.

When an element is obtained, the Brpop command returns two values, namely the key name and the element value. To test the Brpop command, we can open two REDIS-CLI instances in instance a:

Redis A>brpop Queue 0

When you type carriage return, instance 1 is blocked, and an element is added to the queue in instance B:

Redis b>lpush Queue Task (integer) 1

After the Lpush command executes, instance a immediately returns the result:

1) "Queue" 2) "task"

You will also find that the elements in the queue have been taken away:

Redis>llen Queue (integer) 0

In addition to the Brpop command, Redis also provides Blpop,the difference between the Brpop and the Blpop is taken from the left side of the queue when the element is taken from the queue.

Redis Priority Queue

When both tasks are present when sending a confirmation message and sending a notification message (which can be deferred), the former should be performed first. To achieve this, we need to implement a priority queue.

The Brpop command can receive multiple keys at the same time, and its full command format is blpop key [key ...] Timeout, such as Blpop queue:1 Queue:2 0. The meaning is to detect multiple keys at the same time, if none of the keys are blocked, and if one of the keys has an element, the element is ejected from the key. For example, open two REDIS-CLI instances, in instance a:

Redis a>blpop queue:1 queue:2 queue:3 0

In instance B:

Redis b>lpush queue:2 Task (integer) 1

The instance a will return:

1) "Queue:2" 2) "task"

if multiple keys have elements, one element in the first key is taken from left to right。 We first add an element to Queue:2 and Queue:3:

Redis>lpush queue:2 task11) (integer) 1redis>lpush queue:3 task22) Integer 1

Then execute the brpop command:

Redis>brpop queue:1 queue:2 queue:3) "Queue:2" 2) "Task1"

This attribute enables priority-sensitive task queues to be implemented. We use the Queue:confirmation.email and queue:notification.email two keys to store the two tasks of sending a confirmation email and sending a notification message, and then changing the consumer's code to:

Loop$task =brpop Queue:confirmation.email,queue:notification.email,0execute (task[1])

Once the task to send the confirmation message is added to the Queue:confirmation.email queue, regardless of queue:Notification.emailHow many tasks do you have, and consumers will prioritize the task of sending a confirmation email.

Redis Research (15)-Task queue