Solve the order of RABBITMQ message queue and repeat consumption problem

51 period to Korea to play, by the way to a friend company. The so-called nonsense, is the past to listen to technology sharing, there are Python, Golang, devops,docker some topics. On the whole, there are still some gaps between technical aspects and domestic.

Starting with the strong demand for all aspects of the business, we used RABBITMQ as the message queue, using the RABBITMQ ACK mechanism to confirm the reliability of the message.  But RABBITMQ itself is not the absolute message order mechanism, the single queue in multi-consumer can not guarantee its sequencing. Additionally the ACK mechanism triggers the message to be consumed repeatedly, and we need to avoid the problem in design.

The article wrote a little messy, welcome to spray!  In addition, the article continues to update, please go to the original address to view the update. http://xiaorui.cc/?p=4493

About repeated executions of messages

The first thing we can confirm is that the conditions that trigger the repeated execution of the message are very harsh. It is said that the condition will not be triggered in most scenarios ...  It usually occurs when the task times out, or does not return to the state in time, causing the task to re-enter the queue and re-consume.   A disconnect in the RABBTIMQ also triggers the message to be re-queued. The consumer task type is best to support idempotent, and the benefit is that it doesn't matter how many times the task executes, and consumes some performance at most. If you do not support idempotent, such as sending information. Then you need to build a map to record the performance of the task.  Not only success and failure, but also a heartbeat ...    This map can be implemented on the consumer side ... Here will be a problem, there are two consumers C1, C2, a task may be C1 consumption, if again, was C2 executed. So how to know the situation of the task.  Task distribution. The task is made into hash, fixed consumer. Do not try to expand this future in MQ.

In a word, do not guarantee the power of the message, or use a map to record the status of the task.

About the absolute sequential execution of messages

Most of the scenarios we encounter do not require the order of the message, if the message sequence is sensitive, then we give the method is the message body through the hash assigned to the queue, each queue corresponding to a consumer, multi-split queue.  Why do you design this? The same group of tasks will be assigned to the same queue, each queue can have only one worker to consume, so as to avoid the same queue of multiple consumer consumption, the possibility of disorderly ordering. T1, T2 two tasks, T1 although C1 first pop, but it is possible C2 first T2 task to complete.

In a word, take the initiative to assign queues to individual consumers.

Multi-task processing framework based on RABBITMQ

Here, we have implemented a rather complex architecture with RABBITMQ, saving too much MQ connectivity and consumption. Implemented by Python Pika Gevent, because Py has a Gil, so using multi-process to run multicore, the process does not use shared variables, but the queue to pass the ACK signal.

The scenario implemented here is to consume RABBITMQ with Pika, then drop the acquired task into the queue, another process to consume the task, and then trigger the ACK.

END.

http://xiaorui.cc/2017/05/04/%E8%A7%A3%E5%86%B3rabbitmq%E6%B6%88%E6%81%AF%E9%98%9F%E5%88%97%E7%9A%84%E9%A1%BA% e5%ba%8f%e5%8f%8a%e9%87%8d%e5%a4%8d%e6%b6%88%e8%b4%b9%e9%97%ae%e9%a2%98/