RABBITMQ Series III (Deep Message Queuing)

message Persistence isRabbitMQone of the most talked about features,RabbitMQThe biggest mystery is the ability to persist messages on the basis of minimal performance costsRabbitMQthe design of a multilayer message queue. Below, this article is fromMessageQueuethe design and message inMessageQueueA comprehensive overview of the life cycle of twoRabbitMQMessage Queue.

RabbitMQ fully implements the AMQP protocol, similar to a mailbox service. Exchange is responsible for delivering messages to the corresponding message queue based on ExchangeType and Routingkey. Message Queuing is responsible for staging messages before consumers get the messages. In the RabbitMQ ,MessageQueue is mainly composed of two parts, one for amqqueue, is primarily responsible for implementing The logic functions of the AMQP protocol. The other is a backingqueue for storing messages , and this article focuses on the design of backingqueue .

   

in the RabbitMQ in Backingqueue also by 5 A sub-queue consists of: Q1 , Q2 , Delta , Q3 and the Q4 . Once a message in the RabbitMQ enters the queue, it is not fixed, and the state of the message constantly changes as the system's load flows through the queue. There are 5 types of messages in RabbitMQ :

a) Alpha : The contents of the message and the message index are stored in memory;

b) Beta : The message content is saved on disk, and the message index is stored in memory;

c) Gamma : The message content is stored on disk, and the message index is in both disk and memory;

d) Delta : Both the message content and the index are on disk;

Note: For persisted messages, both the message content and the message index must be saved to disk before they are in one of these states, and the Gamma -State message is only persisted when the message is in that state.

     Backingqueuein the5message status in a child queue,Q1and theQ4the corresponding isAlphastatus,Q2and theQ3is aBetastatus,Deltathe corresponding isDeltastate. Above isRabbitMQthe design of the multilayer queue structure, we can see fromQ1to theQ4, the basic experience is theRAMto theDISK, and then toRAMthe design. The advantage of this design is that when the queue load is very high, can save the memory space by saving a part of the message by disk, when the load is reduced, this part of the message gradually back to the memory, by the consumer, so that the entire queue has a good elasticity. Let's look at the workflow for the entire message queue. There are two main factors that cause the flow of the message: one is the consumer gets the message, and the other is because of insufficient memory, which causes the message to be swapped out to disk ( Q1-.>Q2 , Q2->delta , Q3->delta , q4->q3 ). RabbitMQ when the system is running, it calculates the maximum number of messages that can be saved in the current memory based on the speed of the message transmission ( Target_ram_count ), when the number of messages in memory is greater than this value, it causes the message to flow. Messages that enter the queue are usually pressed Q1->q2->delta->q3->q4 , but not every message is bound to go through all the state, depending on the load condition of the system at that time. when the consumer gets the message, it starts with the Q4 queue to get the message if Q4 gets the success, then returns if Q4 is empty, the attempt is from Q3 first, the system will determine Q3 If the queue is empty, and if it is empty, the direct return queue is empty, that is, there is no message in the queue at this time (subsequent argument). If it is not empty, remove the Q3 the message, and then judge at this time Q3 and the Delta The length of the queue, if it is empty, can be considered Q2 , Delta , Q3 and the Q4 All Empty ( Follow-up instructions ) , this will Q1 the message is transferred directly to the Q4 , the next time directly from Q4 gets the message. If Q3 is empty, Delta is not empty, it will Delta to the message in the Q3 , if Q3 non-empty, the next time directly from Q3 gets the message. In the Delta transferred to Q3 in the process, RabbitMQ is read by the index fragment, read a segment first, until the read message is not empty, and then determine the number of messages read and Delta if the number of messages is equal and if they are equal, the Delta is not already in the message, directly Q2 and just read the message into it. Q3 the. If they are not equal, only the message that is read is transferred to Q3 the. This is the process of message flow caused by consumers.         Let's analyze the message swap out due to insufficient memory. The message is swapped out with the number of messages saved in memory+waitACKthe number of messages>target_ram_count. When the condition is triggered, the system first determines if the current entry WaitsACKmessage is faster than the speed at which the message is entered into the queue, the wait is processed firstACKthe message. Steps basicallyQ1->Q2orQ3move, depending onDeltawhether the queue is empty. q4->q3Move,Q2and theQ3toDeltamove.     Finally, let's analyze the two problems left behind, one is why Q3 The queue is empty to determine that the entire team is empty. Imagine if Q3 is empty, Delta not empty, then in Q3 when the last message is removed, Delta the message will be transferred to Q3 on, with Q3 empty Contradiction. If Q2 not empty, then in Q3 Remove the last message if Delta Is empty, the Q2 the message is incorporated into Q3 , with Q3 an empty contradiction. If Q1 not empty, then in Q3 Remove the last message if Delta and the Q3 Is empty, the Q1 the message is transferred to Q4 in, with Q4 an empty contradiction. This also explains another question: why Q3 and the Delta is empty, Q2 Is empty.     This is the whole message RabbitMQ the flow process in the queue. As can be seen from the above process, if the message can be consumed as early as possible, there is no need to go through the process of persistence, as this will add to the overhead of the system. If the message is consumed too slowly, RabbitMQ prevent memory overflow by swapping out the memory.

RABBITMQ Series III (Deep Message Queuing)