Comparison of Kafka, RabbitMQ and ROCKETMQ message middleware--message sending performance--switch from Ali Middleware

Introduction

In distributed system, we widely use message middleware to exchange data between systems, and facilitate asynchronous decoupling. Now open source message middleware has a lot of time ago our own product ROCKETMQ (Metaq core) also smooth open source, get everyone's attention.

So, what is the performance of message middleware?

With this in doubt, our middleware test group performs a performance comparison of the common three types of message products (Kafka, RabbitMQ, ROCKETMQ).

Kafka is the LinkedIn Open source distributed publish-Subscribe messaging system, currently owned by the Apache grading project. The main feature of Kafka is the pull-based mode to handle message consumption, the pursuit of high throughput, the first goal is to use for log collection and transmission. Version 0.8 began to support replication, does not support transactions, the duplication of messages, loss, errors are not stringent requirements, suitable for generating large amounts of data for the data collection business of Internet services.

RABBITMQ is an open source Message Queuing system developed using the Erlang language, which is implemented based on the AMQP protocol. The main features of AMQP are message-oriented, queue, routing (including point-to-point and publish/subscribe), reliability, and security. The AMQP protocol is more used in enterprise systems, where the requirements for data consistency, stability, and reliability are high, and the performance and throughput requirements are second.

ROCKETMQ is Ali open source of the message middleware, it is pure Java development, with high throughput, high availability, suitable for large-scale distributed system application features. ROCKETMQ idea originated from Kafka, but not a copy of Kafka, it is the reliable transmission and transactional optimization of the message, at present, Ali Group is widely used in trading, recharge, flow calculation, message push, log streaming, Binglog distribution and other scenarios. Test Purpose

Compares the performance of Kafka, RabbitMQ, ROCKETMQ sending small messages (124 bytes). We only focus on the performance of the service side, so the pressure measurement standards are:

Increasing the pressure on the sending side until the system throughput no longer increases and the response time is stretched. At this point the server has a performance bottleneck, and can obtain the appropriate system throughput. Test Scenario

In the synchronous sending scenario, the performance of the three message middleware is clearly differentiated:

Kafka throughput of up to 17.3w/s, is worthy of high-throughput message middleware industry leader. This depends primarily on its queue pattern, which guarantees that the process of writing the disk is linear IO. At this point, broker disk IO has reached a bottleneck.

ROCKETMQ also performed well, with throughput at 11.6w/s and disk IO%util approaching 100%. The ROCKETMQ message is returned with an ACK after it is written to the memory, and a separate thread is dedicated to the operation of the brush disk, and all messages are sequential write files.

RABBITMQ Throughput 5.95w/s,cpu resource consumption is high. It supports the AMQP protocol, which is very heavyweight, to ensure that the reliability of the message is a trade-off in throughput. We also did the RABBITMQ performance test in the message persistence scenario, with throughput around 2.6w/s. Test Conclusion

On the server side to handle the performance of synchronous send, KAFKA>ROCKETMQ>RABBITMQ. Appendix: test Environment

The server is a stand-alone deployment with the following machine configuration:

App version:

Test Scripts

not to be continued

Before we compare the simplest small message sending scenario, Kafka temporarily wins. In practical applications, however, this pattern of pure accumulation without consumption is very rare. In the Internet application scenario, the message consuming ability of message middleware and the support ability of mass subscribers are especially important.

Next we will focus on the number of partitions, message size, consumption patterns and other factors, the three types of message middleware contrast. If you are interested in this series of articles, click on the "Attention" bar, please look forward to the follow-up report!

Reprinted from  https://yq.aliyun.com/articles/62831?spm=a2c4e.11163080.searchblog.82.5df32ec1obx4zt