Producer Traffic Control for Message Queue rabbitmq and activemq

20120825 Zheng

Q: Why do MQ need to control the producer traffic? A: The trouble is: "The implementation and design of virtual machines like Erlang have not prevented users from throwing messages to the Message Queue of a process. When the message production speed is too fast, when the processing capacity of the process is exceeded, these messages are accumulated, occupying more memory and eventually causing VM crash. 』 Q: Why do I need to know that MQ is implementing producer traffic control? A: When you find that your producers are suspended or blocked, you need to know whether to adjust the consumption rate of consumer or call Memory threshold of MQ. 1. rabbitmq 2.8.0 + Traffic Control

Rabbitmq 2.8.0 + introduces a new feature "Internal Flow Control ".

So far, rabbitmq has three types of traffic control: 1.1. Per-connection Flow Control Traffic Control for each connection. That is, rabbitmq will actively block connections that publish messages too quickly (connections) without any configuration. If the connection is blocked, a blocked status is displayed on the rabbitmqctl console. Rabbitmq's traffic control mechanism is based on the Credit congestion control mechanism. Later Zheng will list the details of this old control mechanism in Appendix. 1.2. Memory-based Flow Control Rabbitmq checks the physical memory value of the machine at startup. By default, when MQ occupies more than 40% of the memory, MQ will actively throw a memory warning and block all connections (connections ). You can also modify Rabbitmq. config File to adjust the memory threshold. The default value is 0.4, as shown below:

 
[{Rabbit, [{vm_memory_high_watermark, 0.4}].
 

When MQ is started, the memory threshold value is written  Rabbitmq_nodename . Log File, as follows:

 
Memory limit set to 2048 MB.
 

If the MQ server cannot recognize your system or you are using a Windows system, it will write a warning message Rabbitmq_nodename . Log File, as follows:

 
= Warning report === 29--200-2009: 17: 23: 44 === unknown total memory size for your OS {UNIX, magic_homebrew_ OS}. Assuming memory size is 1024 MB.
 

 

1.3. disk-based Flow Control By default, if the remaining disk space is less than 1 GB, rabbitmq actively blocks all producers. This threshold value is also adjustable. Ii. Traffic Control of Apache activemq 2.1. activemq's producer traffic control trigger conditions include:

• Whether or not MQ has a persistent Configuration:
• ! Memory Used by activemq reaches MemoryusageConfiguration value. The default value is 64 MB;
• If MQ is configured persistently:
• ! The usecache switch is enabled, indicating that persistent messages are cached for quick access. The default value is true;
• ! The total number of messages cached in memory reachesMemorylimitConfiguration value. The default value is 1 MB.

2.2. triggered: Stuck gets stuck when the producer calls the message sending function ). (Note: keywords activemq + stuck + producer or activemq + block + producer) Appendix

I. based on letters of credit(Credit)Congestion Control Mechanism

In 1993, the ATM field began to look for a transmission mechanism that could dynamically allocate bandwidth and prevent cell loss. Therefore, an API (Available Bit Rate, Available Bit Rate) Service is proposed. An API does not force the network to allocate a fixed bandwidth, and the network dynamically adjusts the bandwidth allocated to each API connection through feedback. Many experts have invested in research on the congestion control specifications of the ABR service.

Credit-based solutions developed by Harvard University H.t. Kung The professor first proposed to adopt a link-level Flow Control Mechanism with a single link or virtual circuit VC As the basic control unit. Each link has a cell sending node (which can be a source or exchange node) and a receiving node (which can be an exchange node or a destination system). Each node is VC Maintains a queuing queue, and the cell receiving end monitors each entry. VC Determines the length of the queue that the sender can send. VC Number of cells on (notified by letter of credit), the cell sender can only send the maximum number of cells allowed by the letter of credit value, if there is only one VC , The value of the letter of credit should be large enough to make full use of the link bandwidth. Generally, the value of a letter of credit must meet the following conditions: Greater than or equal   Link cell rate Multiply Link to return transmission latency.

Figure6-17The basic working principle of the credit-based congestion control mechanism is given. The cell receiver first sends a letter of credit to the cell sender to notify the available buffer capacity. After receiving the letter of credit, the sender can determine the number of cells to be sent. 

Its biggest disadvantage is its complexity. It needs to maintain this letter of credit mechanism between each adjacent node and increase the delay of cells.

2. How is rabbitmq implemented?

"In essence, rabbitmq is used to monitor the mailbox of each process. When the load of a process is too high to receive messages, the mailbox of this process begins to pile up messages.

When accumulated to a certain amount, it will block the upstream process and prevent it from receiving new messages. In this way, the mailbox of the upstream process also begins to backlog messages.

When a certain amount of data is reached, the upstream process will be blocked to receive messages. At last, the process responsible for receiving network data packets will be blocked and data reception will be suspended.

This is a bit like a multi-level reservoir. When the downstream reservoir is under too much pressure, the upstream reservoir has to close the gate, so that its pressure is getting bigger and bigger. You need to close the upstream reservoir gate, until the most upstream gate is closed. 』 --Http://ybbct.iteye.com/blog/

References 1, http://www.rabbitmq.com/memory.html,RabbitMQ flow control2, internal, Analysis of rabbitmq performance (persistence) bottleneck through traffic control mechanism 5, http://activemq.apache.org/producer-flow-control.html6,http://working-with-activemq.blogspot.com/2012/05/performance-improvements.html 7, http://stackoverflow.com/questions/5291679/activemq-topic-flooding8,http://www.huaishao8.com/tag/activemq