Performance issues with multiple consumer processing messages at the same time in SPRING+ACTIVEMQ

Recently in the work of data docking, with the use of ACTIVEMQ, I need to receive messages from the ACTIVEMQ and processing, but I process the data is a little more complex, gradually in the message queue heap of data more and more, I think of my side to open a few threads to process the message.

However, you will find that the server occupies an unusually high network bandwidth, careful analysis found that MQ queue when there is no unusually high network traffic, only when the team will generate very high network traffic. The final discovery is the problem caused by the jmstemplate of spring and the prefetch mechanism of ACTIVEMQ. Research source Discovery jmstemplate Implementation mechanism is: each call to receive () will create a new consumer object, run out to destroy . Under normal circumstances, the resource cost of duplicating the creation of consumer is wasted, and the network traffic that is 10 times times the normal situation is not generated. But ACTIVEMQ has a mechanism to lift high performance prefetch, there will be serious problems at this time.

The prefetch mechanism of ACTIVEMQ:
Each time consumer is connected to MQ, MQ stores many message-to-consumer (provided there is a large number of messages in MQ), and the number of pre-storage of the message depends on Prefetchsize (default is $). The purpose of this mechanism is obviously to allow client-side code to repeatedly receive operations with a consumer, which can significantly improve the performance of the team.

This mechanism has a serious problem with jmstemplate, and each time jmstemplate.receive () generates 1000 messages of network traffic, but because Jmstemplae does not reuse consumer, Results in the subsequent 999 messages being discarded. Repeated jmstemplate.receive (), the surface can not see any problem, in fact, network bandwidth will cause a lot of waste .

Solution Solutions

1, if insist on using jmstemplate, need to set prefetch value of 1, equivalent to disable the ACTIVEMQ prefetch mechanism , at this time feel the most robust, even if multiple threads, repeatedly call Jmstemplate.receive () There will be no problem. However, there is a waste of resources because it is necessary to create consumer repeatedly and communicate with the server frequently, but it is not a problem in applications with low performance requirements.

2, do not use Jmstemplate, manually create a consumer, at this time can make full use of prefetch mechanism. In conjunction with multithreading, each thread has its own consumer, which maximizes the speed advantage of MQ at high throughput.

PS: How to see the prefetch mechanism of ACTIVEMQ?

The default value of Prefetchsize is 1000, so as long as the message in the queue is less than 1000, the consumer regardless of the number of threads, we will find that the actual processing of the message thread is always only one, the other threads are actually working hard, can not help.

Performance issues with multiple consumer processing messages at the same time in SPRING+ACTIVEMQ