Redis Research (15th)-task queue, redis research task queue

Redis Research (15th)-task queue, redis research task queue

During website development, page rendering will be blocked when pages require long-time operations such as sending emails and complex data operations. To prevent users from waiting too long, separate threads should be used to perform such operations. However, some programming languages or frameworks are not easy to implement multithreading. At this time, it is easy to think of other processes for implementation. Imagine that a process can complete the mail function. On the page, you only need to find a way to notify the process to send an email to the specified address.

The notification process can be implemented through the task queue. As the name implies, a task queue is a queue for transferring tasks ". There are two types of entities that interact with task Queues: producer and consumer ). The producer puts the tasks to be processed into the task queue, while the consumer continuously reads and executes the task information from the task queue.

For the mail operation, the page program is the producer, and the mail process is the consumer. When an email needs to be sent, the page program assembles the email address, subject, and body into a task and stores it in the task queue. At the same time, the mail process will constantly check the task queue. Once a new task is found, it will be taken out of the queue and executed

Line. Thus, inter-process communication is realized.

The following benefits apply to task Queues:
(1) loose coupling. The producer and consumer do not need to know the implementation details of each other, but only need to specify the task description format. This allows producers and consumers to be written in different programming languages by different teams.

(2) It is easy to expand. Consumers can have multiple servers, which can be distributed across different servers. This can easily reduce the load of a single server.

Next we use Redis to implement task queue

The list type of Redis. Use the LPUSH and RPOP commands to implement the queue. To implement the task queue, you only need to ask the producer to add the task to a key using the LPUSH command. On the other side, let the consumer continuously use the RPOP command to retrieve the task from the key.

In the preceding example, you need to know the recipient address, subject, and body of the email before you complete the mail task. Therefore, the producer needs to make up these three information objects and serialize them into strings, and then add them to the task queue. The consumer cyclically pulls tasks from the queue, like the following pseudo code:

# Infinite loop read the content in the task queue loop $ task = RPOR queueif $ task # execute it if there is a task in the task queue ($ task) else # If not, wait 1 second to avoid too frequent requests for data wait 1 second

This is a simple task queue implemented by using Redis. But there is still a bit imperfect: when there are no tasks in the task queue, the consumer calls the RPOP command once per second to check whether there are new tasks. It would be nice to notify the consumer once a new task is added to the task queue. In fact, this requirement can be achieved through the BRPOP command.

The BRPOP command is similar to the RPOP command. The only difference is that when there are no elements in the list, the BRPOP command will block the connection until new elements are added. The code above can be rewritten:

Loop # if there are no new tasks in the task queue, the BRPOP command will be blocked and execute () will not be executed (). $ Task = BRPOP queue, 0 # The returned value is an array (see the Introduction below), and the second element of the array is the task we need. Execute ($ task [1])

The BRPOP command receives two parameters. The first parameter is the key name, and the second parameter is the timeout time, in seconds. If no new element is obtained after this time, nil is returned. In the preceding example, the timeout value is "0", indicating that the waiting time is not limited. That is, if no new element is added to the list, it will be blocked forever.

After an element is obtained, the BRPOP command returns two values: the key name and the element value. To test the BRPOP command, we can open two redis-cli instances in instance:

redis A＞BRPOP queue 0

After you press enter, instance 1 will be blocked. Then, an element is added to queue in instance B:

redis B＞LPUSH queue task(integer) 1

After the LPUSH command is executed, instance A immediately returns the result:

1) "queue"2) "task"

At the same time, we will find that the elements in the queue have been removed:

redis＞LLEN queue(integer) 0

In addition to the BRPOP command, Redis also provides BLPOP. The difference between BLPOP and BRPOP is that BLPOP will be taken from the left of the queue when it gets elements from the queue.

Redis priority queue

When both the send confirmation email and the send notification email (delayed) Tasks exist at the same time, the former should be executed first. To achieve this goal, we need to implement a priority queue.

The BRPOP command can receive multiple keys at the same time. The complete command format is BLPOP key [key…]. Timeout, such as BLPOP queue: 1 queue: 2 0. It indicates that multiple keys are detected at the same time. If no key has any element, blocking is performed. If one key has an element, an element pops up from the key. For example, open two redis-cli instances in instance:

redis A＞BLPOP queue:1 queue:2 queue:3 0

In instance B:

redis B＞LPUSH queue:2 task(integer) 1

Instance A will return:

1) "queue:2"2) "task"

If multiple keys have elements, take one of the first keys from left to right. Add one element to each of queue: 2 and queue: 3:

redis＞LPUSH queue:2 task11) (integer) 1redis＞LPUSH queue:3 task22) integer) 1

Then run the BRPOP command:

redis＞BRPOP queue:1 queue:2 queue:3 01) "queue:2"2) "task1"

This feature enables task Queues with different priorities. We use the queue: confirmation. email and queue: notification. email keys respectively to store two types of tasks: Send confirmation mail and send notification mail, and then change the consumer code:

loop$task =BRPOP queue:confirmation.email,queue:notification.email,0execute(task[1])

Once the task of sending the confirmation email is added to the queue: confirmation. email queue, the consumer will first complete the task of sending the confirmation email regardless of the number of tasks in queue: notification. email.