Spring Boot implements RabbitMQ deferred consumption and deferred retry queue

This article mainly extracts from: detailed introduction Spring Boot + RABBITMQ Implementation delay queue

And added some of their own understanding, recorded, for later inspection.

Project Source:

• Spring-boot-rabbitmq-delay-queue implementation
• Stream-rabbitmq-delay-queue implementation

What is the delay queue in the background?

As the name implies, a deferred queue is a queue in which messages that enter the queue are deferred for consumption. In the general queue, once the message is queued, it will be consumed by the consumer immediately.
What can the delay queue do? The delay queue is used more for scenarios that require delayed work. The most common are the following two scenarios:

• delayed consumption . For example: After the user generated the order, it will take a period of time to check the payment status of the order, if the order is still not paid, the need to close the order in a timely manner, after a successful user registration, after a period of time, such as a week after verifying the user's usage, if the user is found to be less active, send an
• Delay retry . For example, consumers fail to consume messages from the queue, but want to retry automatically after a period of delay.

If the delay queue is not used, then we can only do it through a polling scanner. This scheme is neither elegant nor convenient for a unified service to facilitate the use of developers. But with the delay queue, we can do it easily.

Implementation ideas

Before introducing concrete implementation ideas, let's introduce the two features of RABBITMQ, one is time-to-live Extensions and the other is dead letter exchanges.

Time-to-live Extensions

RABBITMQ allows us to set TTL (Time to live) for a message or queue, which is the expiration date. The TTL indicates the maximum time a message can survive in a queue, in milliseconds. That is, when a message is set to TTL or when a message enters a queue with a TTL set, the message "Dies" after TTL seconds, becoming dead letter. If both the TTL of the message is configured and the TTL of the queue is configured, the smaller value is taken. For more information, please refer to the official documentation.

Dead Letter Exchange

As mentioned earlier, the message that was set to TTL will become dead letter after it expires. In fact, in RABBITMQ, a total of three kinds of message "death" form:

• The message was rejected. By calling Basic.reject or Basic.nack and setting the Requeue parameter to False.
• The message expired because the TTL was set.
• The message enters a queue that has reached its maximum length.

If the queue has dead Letter Exchange (DLX) set, these dead letter will be publish back to dead letter exchange and routed to other queues via dead letter exchange. For more information, please refer to the official documentation.

Flow chart

Smart you must have figured out how to combine the TTL and DLX features of RABBITMQ to implement a delay queue.

For the two scenarios of the above delay queue, we have the following two flowchart:

Deferred consumption

Deferred consumption is the most common usage pattern for delay queues. As shown, the producer-generated message first enters the buffer queue (the red queue in the figure). With the TTL extension provided by RABBITMQ, these messages are set to expire, that is, the time of the consumption delay. After the message expires, these messages are forwarded to the actual consumption queue (the blue queue in the graph) through the configured DLX to achieve the effect of deferred consumption.

Deferred retry

Delay retry is essentially a deferred consumption, but the structure of this model and the normal delay consumption of the flowchart is more different, so separate to carry out the introduction.

As shown, the consumer finds that the message processing has an exception, such as an exception due to network fluctuations. If you do not wait for a period of time, and then try again, it is likely to lead to the failure during this period, resulting in a certain amount of wasted resources. Then we can put the its first in the buffer queue (the red queue in the picture), and so on after a period of delay after the message into the actual consumption queue (the blue queue in the picture), at this time due to the "longer" period, some of the abnormal fluctuations are usually restored, these messages can be normal consumption.

Code implementation Configuration Queue

From the flowchart above, we can see that a delay queue implementation requires a buffer queue and an actual consumption queue. And because in RABBITMQ, we have two ways to configure message expiration, so in our code, we've configured three queues altogether:

• The Delay_queue_per_message_ttl:ttl configures the buffer queue on the message.
• The Delay_queue_per_queue_ttl:ttl configures the buffer queue on the queue.
• Delay_process_queue: Actual consumption queue.

We configure the above queue as a bean in the way of Java CONFIG. Since we have added spring-boot-starter-amqp extensions, Spring boot will automatically create these queues at startup based on our configuration. To facilitate the next test, we configured the Delay_queue_per_message_ttl and Delay_queue_per_queue_ttl DLX to be the same, and expired messages are forwarded through DLX to Delay_process_ Queue

Delay_queue_per_message_ttl

First introduce the configuration code for DELAY_QUEUE_PER_MESSAGE_TTL:

@BeandelayQueuePerMessageTTL() {    return QueueBuilder.durable(DELAY_QUEUE_PER_MESSAGE_TTL_NAME)                       .withArgument("x-dead-letter-exchange"// DLX，dead letter发送到的exchange                       .withArgument("x-dead-letter-routing-key"// dead letter携带的routing key                       .build();}

Where the name of the DLX that the dead letter in the x-dead-letter-exchange queue is forwarded to is declared, and the x-dead-letter-routing-key Routing-key name that these Badmail carries when forwarding.

Delay_queue_per_queue_ttl

Similarly, Delay_queue_per_queue_ttl's configuration code:

@BeandelayQueuePerQueueTTL() {    return QueueBuilder.durable(DELAY_QUEUE_PER_QUEUE_TTL_NAME)                       .withArgument("x-dead-letter-exchange"// DLX                       .withArgument("x-dead-letter-routing-key"// dead letter携带的routing key                       .withArgument("x-message-ttl"// 设置队列的过期时间                       .build();}

The configuration of the Delay_queue_per_queue_ttl queue is one more than the configuration of the Delay_queue_per_message_ttl queue x-message-ttl , which is used to set the queue expiration time.

Delay_process_queue

The Delay_process_queue configuration is the simplest:

@BeandelayProcessQueue() {    return QueueBuilder.durable(DELAY_PROCESS_QUEUE_NAME)                       .build();}

Configure Exchange Configuration DLX

First, we need to configure DLX with the following code:

@BeandelayExchange() {    returnnewDirectExchange(DELAY_EXCHANGE_NAME);}

The DLX is then bound to the actual consumption queue, which is delay_process_queue. All dead-letter will then be forwarded to Delay_process_queue via DLX:

@BeandlxBinding(Queue delayProcessQueue, DirectExchange delayExchange) {    return BindingBuilder.bind(delayProcessQueue)                         .to(delayExchange)                         .with(DELAY_PROCESS_QUEUE_NAME);}

Configure the exchange that is required for deferred retries

We can see from the flowchart of the deferred retry that we need to forward the message to the buffer queue after the message processing has failed, so the buffer queue also needs to bind an exchange. In this example, we use the Delay_process_per_queue_ttl as the buffer queue in the deferred retry.

Define Consumer

We create the simplest consumer processreceiver, the consumer who listens to the Delay_process_queue queue and, for the messages received, will:

• If the message body is not equal to Fail_message, then he will output the message body.
• If the message body in the message happens to be fail_message, then he simulates throwing an exception, and then redirects the message to the buffer queue (corresponding to the deferred retry scenario).

In addition, we need to create a new monitoring container to store the consumer, the code is as follows:

@BeanprocessContainer(ConnectionFactory connectionFactory, ProcessReceiver processReceiver) {    newSimpleMessageListenerContainer();    container.setConnectionFactory(connectionFactory);    container.setQueueNames// 监听delay_process_queue    container.setMessageListener(newMessageListenerAdapter(processReceiver));    return container;}

Now that our pre-configured code is all written, we need to write a test case to test our delay queue.

Writing test cases to defer consumption scenarios

First we write the test code used to test the TTL setting on the message.

We use spring-rabbit the Rabbittemplate class provided under the package to send the message. Since we have added spring-boot-starter-amqp extensions, Spring boot automatically loads rabbittemplate as beans into the container at initialization time.

Solve the problem of sending messages, then how to set the TTL for each message? Here we need to use messagepostprocessor. Messagepostprocessor is typically used to set the header of a message and the properties of a message. We create a new Expirationmessagepostprocessor class to be responsible for setting the TTL property of the message:

/**   * Set message expiration time   */ public  class  Expirationmessagepostprocessor implements  messagepostprocessor {private< /span> final  Long ttl; //Ms  public  expirationmessagepostprocessor  (Long TTL) {this . ttl  = ttl; }  @Override  public  Message postprocessmessage  (Message message) throws  amqpexception {message. (). setexpiration  (TTL.) tostring  ()); //setting the Per-message expiration time  return  message; }}

Then, when you call the Convertandsend method of Rabbittemplate, you can pass in the expirationmessagepostporcessor. We send 3 messages to the buffer queue with an expiration time of 1 seconds, 2 seconds, and 3 seconds. The specific code is as follows:

@Test Public void Testdelayqueuepermessagettl()throwsinterruptedexception {processreceiver.latch=NewCountdownlatch (3); for(inti =1; I <=3; i++) {LongExpiration = i * +; Rabbittemplate.Convertandsend(Queueconfig.Delay_queue_per_message_ttl_name, (Object) ("Message from Delay_queue_per_message_ttl with expiration"+ expiration),New Expirationmessagepostprocessor(expiration)); } processreceiver.latch.await();}

A careful friend must ask, why add a countdownlatch to the code? This is because if there is no latch blocking the test method, the test is closed directly, the program exits, we can not see the message is delayed consumption performance.

So similarly, the Code for the test TTL setting on the queue is as follows:

@TestpublicvoidtestDelayQueuePerQueueTTLthrows InterruptedException {    ProcessReceiver.latchnew CountDownLatch(3);    for (int13; i++) {        rabbitTemplate.convertAndSend(QueueConfig.DELAY_QUEUE_PER_QUEUE_TTL_NAME,                "Message From delay_queue_per_queue_ttl with expiration " + QueueConfig.QUEUE_EXPIRATION);    }    ProcessReceiver.latch.await();}

We send 3 messages to the buffer queue. Theoretically, these 3 messages will expire in 4 seconds.

Deferred retry scenario

We also need to test the deferred retry scenario.

@TestpublicvoidtestFailMessagethrows InterruptedException {    ProcessReceiver.latchnew CountDownLatch(6);    for (int13; i++) {        rabbitTemplate.convertAndSend(QueueConfig.DELAY_PROCESS_QUEUE_NAME, ProcessReceiver.FAIL_MESSAGE);    }    ProcessReceiver.latch.await();}

We send 3 messages to delay_process_queue that will trigger fail, in theory the 3 messages will be automatically retried after 4 seconds.

I understand.

delay the consumption process (each message can be set to a separate expiration time):

• 1. Declare the Delay_queue_per_message_ttl queue: Dead-letter queue, set the DLX parameter, and include X-dead-letter-exchange for incoming exchange after failure (value Delay_ Exchange, the actual consumption switch), X-dead-letter-routing-key represents the failed routing key (the value is Delay_process_queue, which is the actual consumption queue).
• 2. Declare the Delay_process_queue queue: The actual consumption queue.
• 3. Declare the delay_exchange switch: The actual consumption switch, the type is Direct (one by one corresponds).
• 4. Declare dlx_binding binding: Bind the actual consumption queue to the actual consumption switch (the Routing key Rule value is Delay_process_queue).
• 5. Publish a message, the routing key is delay_queue_per_message_ttl (sent to the dead letter queue), and the expiration time for each message is set separately through the header: When the expiration time takes effect, the message goes to the actual consumption queue.
• 6. Declare a consumer, listen to the Delay_process_queue queue (that is, the actual consumption queue): The message is normally consumed, to achieve the purpose of deferred consumption.

deferred consumption process (all messages uniformly set expiration time):

• 1. Declare the Delay_queue_per_queue_ttl queue: Dead-letter queue, set the DLX parameter, and include X-dead-letter-exchange for incoming exchange after failure (value Delay_ Exchange, the actual consumption switch), the X-dead-letter-routing-key represents the failed routing key (the value is Delay_process_queue, which is the actual consumption queue), and the X-message-ttl indicates the queue message expiration time.
• 2. Declare the Delay_process_queue queue: The actual consumption queue.
• 3. Declare the delay_exchange switch: The actual consumption switch, the type is Direct (one by one corresponds).
• 4. Declare dlx_binding binding: Bind the actual consumption queue to the actual consumption switch (the Routing key Rule value is Delay_process_queue).
• 5. Publish a message with the routing key Delay_queue_per_queue_ttl (sent to the dead-letter queue): When the expiration time takes effect, the message goes to the actual consumption queue.
• 6. Declare a consumer, listen to the Delay_process_queue queue (that is, the actual consumption queue): The message is normally consumed, to achieve the purpose of deferred consumption.

Deferred retry process :

• 1. declares the Delay_process_queue queue: The actual consumption queue.
• 2. declares the Delay_queue_per_queue_ttl queue: Dead letter queue, setting the DLX parameter, which contains a x-dead-letter-exchange that is entered after failure Exchange (value Delay_exchange, actual consumption switch), X-dead-letter-routing-key represents the failed routing key (the value is Delay_process_queue, which is the actual consumption queue), X-message-ttl indicates the queue message expiration time.
• 3. declares the delay_exchange switch: The actual consumption switch, the type is Direct (one by one corresponds).
• 4. declares the per_queue_ttl_exchange switch: Dead-letter switch, type Direct (one by one corresponds).
• 5. declares dlx_binding binding: binds the actual consumption queue to the actual consumption switch (the Routing key Rule value is Delay_process_queue).
• 6. declares queue_ttl_binding binding: The dead-letter queue and the dead-letter switch are bound (the Routing key Rule value is Delay_queue_per_queue_ttl).
• 7. publishes a message with a routing key of Delay_process_queue (sent to the actual consumption queue).
• 8. declares a consumer, listens to the Delay_process_queue queue (that is, the actual consumption queue): The consumer monitors the message, when an exception occurs during processing, the message is resent to the private messages queue, Then wait for the expiration time to take effect, the message is then transferred to the actual consumption queue, re-consumption, to achieve the purpose of deferred retry.

Note: In the process of delaying consumption, we did not create a dead-letter switch, then why can we publish the message? The reason is that RabbitMQ uses the default Exchange and creates a default Binding (type Direct), and the results are visible through the rabbitmqadmin list bindings command.

Spring Cloud Stream RabbitMQ DLX implementation: _rabbitmq_consumer_properties

• _rabbitmq_consumer_properties ( official website )
• Build message-driven microservices with Spring Cloud Stream
• RABBITMQ Publish subscription combat-Delay Retry Queue implementation
• Detailed description of Spring Boot + RABBITMQ Implementation delay Queue ( recommended )

Spring Boot implements RabbitMQ deferred consumption and deferred retry queue