Open source VS. Business, messaging middleware things you don't know.

November 23, the new Torch Network middleware technical expert Tuo teacher in the dba+ Community Middleware user Group has a theme for "open source VS business, message middleware You do not know of those things," the online share. Small series specially sorted out the essence of the content, for everyone to learn to communicate.

Guest profile

• New Torch Network Middleware Technical Experts

• Worked at IBM South China GTS for 4 years, IBM WebSphere, MQ, CICS product Line Technical Experts

• 5 Mobile operators (Guangdong Mobile, Zhejiang Mobile) operations and maintenance experience, 3 years of Java development and after-sales experience

Lecture transcript

With the rise of cloud computing, the popularity of Docker and microservices, distributed Message Queuing technology has become an indispensable component of cloud computing platform. Today, let me share with you the differences between the mainstream business and open source messaging middleware in the market.

What is Messaging middleware MOM (message oriented middleware) leverages efficient and reliable messaging mechanisms for platform-independent data exchange and integrates distributed systems based on data communication. By providing a messaging and Message Queuing model, it can extend communication between processes in a distributed environment.

• Asynchronous: The method of asynchronous communication based on store-and-forward mechanism, the sender sends the message to the message server, the message server stores the message in several queues, and forwards the message to the receiver at the appropriate time.

• Loose coupling: The life cycle of the customer and service object is loosely coupled, and the lifecycle of the message queue and service is guaranteed by MOM, which does not have to be the same, that is, the receiver does not necessarily run when the message is sent, and the sender does not necessarily run when the message is received.

• Reliability: The message is guaranteed by MOM that messages are not lost because of a network connection exception, a process exception restart, and so on.

The messaging pattern is divided into PTP and pub/sub two modes.

PTP (Point–to-point)

• There is only one consumer for each message.

• The sender and receiver have no time dependence (no timing dependencies).

• Recipient successfully receives response mechanism.

Pub/sub (Publish/subscribe)

• Each message can have more than one subscriber.

• The sender and the recipient must synchronize, and the client receives the message only after the subscription.

Message Middleware has five applicable business scenarios: asynchronous notification, data replication, log synchronization, delay queue, broadcast notification.

• Asynchronous notification: Provides distributed transaction support for service-oriented architecture (SOA), and ensures global data consistency. such as order Processing scheduling notification, order disassembly after sending open, billing, accounting and other due reminders, to inform SMS, order status, two times confirmation.

• Data replication: The use of message middleware to copy data from source to multiple destinations, to meet the needs of search, offline analysis and changes in table rules. such as inter-system TF data Synchronization Multi-system, bureau data increment release and so on.

• Log synchronization: The application synchronizes logs to message middleware reliably and asynchronously, and can perform real-time or offline analysis of the logs. such as unified log Platform Center log storage and so on.

• Delay queue: The message middleware as a reliable delay queue, distributed environment of the timer. such as receiving order storage, System log storage and other high-frequency db write scene.

• Broadcast notifications: Reliable in-cluster broadcast notifications, for events such as notifying cache invalidation. such as product cache refresh.

Message Protocol JMS VS AMQP

JMS provides two message models, Peer-2-peer (Point-to-point) and Publish-subscribe (publish subscription) models. In JMS, message routing is very simple, with producer and consumer linked to the same queue (peer-to) or topic (PUB/SUB) for message routing. Jmsconsumer supports message selector (message selectors, through message selectors, consumer can consume only those messages that have been filtered by selector.

AMQP is a protocol, more precisely a binary Wire-level protocol (link protocol). This is the essential difference between it and JMS. AMQP does not qualify from the API layer, but instead directly defines the data format of the network interchange. This makes it possible to realize the provider nature of AMQP is cross-platform.

In AMQP, there are some differences between message routing (messagerouting) and JMS, and the role of exchange and binding is increased in AMQP. Producer sends a message to exchange,binding the message that determines exchange should be sent to that queue, and consumer consumes the message directly from the queue. The queue and Exchang bind have consumer to decide.

The current mainstream messaging middleware in the market is IBM MQ, WebLogic JMS, ActiveMQ, Rabbit MQ, Rocket MQ, Apollo, etc.

• IBM Mq:websphere MQ is the IBM Business Integration Foundation product, with its unique security mechanism, simple and fast programming style, high stability, scalability and cross-platform, as well as strong transaction processing and messaging capabilities, become the industry's most market share of the message middleware products.

• WebLogic Jms:weblogic JMS is a high performance, clustered messaging Server for Oracle, based on WebLogic products that support database and file persistence and fully support XA transactions.

• ROCKETMQ:ROCKETMQ is Ali open source distributed, queue model of message middleware, support strict message sequence, support topic and queue two modes, billion-level message stacking ability, more friendly distributed features, and support push and pull mode consumption messages.

• ACTIVEMQ:ACTIVEMQ is the most popular open source messaging and integration Server on the market today. It delivers very fast messaging, supports multiple cross-platform clients and protocols, and makes it easy to build enterprise-class integration patterns while supporting the JMS1.1 and j2ee1.4 specifications. ACTIVEMQ is based on Apache2.0 license.

• Based on the ACTIVEMQ prototype, Apollo:apollo is a faster, more reliable, and easier to maintain message agent tool, and Apache says Apollo is the fastest and most robust stomp (streaming Text orientated message Protocol, Stream text-directed message protocol) server.

• RABBITMQ:RABBITMQ is a Message Queuing product implemented with Erlang by the AQMP protocol, which implements the broker schema, which can be queued on the central node before it is sent to the client. This feature makes RABBITMQ easy to use and deploy, and is suitable for many scenarios such as routing, load balancing, or message persistence.

This is the final result of the comparison of the functional and performance of each product of a team, from the results:

Activemq Each function module is relatively perfect, does not support the cluster console management, in the subscription mode performance test (10K and the following message size) leading other products.

IBM WMQ has the most complete feature point implementations, leading other products in point-to-point mode read-write hybrid performance testing.

Oracle JMS is a weblogic component, a non-standalone product with a relatively complete set of functions, leading other products in point-to-point mode read-only performance testing, and does not support the AMQP message protocol.

The RABBITMQ has a unique mirroring queue feature that supports the persistence of distributed memory replication and has the advantage of a distributed extension, which is rich in performance monitoring for the console.

ROCKETMQ function Module Support Poor, the implementation of the message protocol is a custom text transfer protocol, in the 1M message size of the performance test leading other products.

The Apollo product is not mature enough to support cluster mode (message routing).

IBM WMQ products in commercial products have high maturity, complete function, wide application, strong compatibility, good cross-platform and cross-language support. Oracle JMS is a WebLogic-based component that supports only JMS and WebLogic T3 message protocols and does not support the AMQP protocol. Cross-platform support needs to be perfected. ACTIVEMQ is the most mature and most functional in open source products.

Thematic analysis of basic functions

• Overall, the product is relatively good for the C + + language Client API and message priority.

• ROCKETMQ does not support user name and password authentication, dead letter mechanism, message expiration function.

Basic features-language and protocol support

Basic function-User name password authentication: mainly through the new user name and password, the use of JMS Client connection attempt to achieve a user name password inconsistency is allowed to connect.

• The IBM WMQ is comprised of queue managers, queues, and channels, and each component can be accessed through the interface.

• Oracle JMS Message Queuing is based on AdminServer instances, which add users through the interface and specify access control for JMS message queues.

• RABBITMQ is access control based on the different virtual hosts of the business division.

• ROCKETMQ does not support user name password authentication.

• Activemq/apollo supports access control for both the topic and queue modes.

Basic functionality-dead-letter mechanism: primarily by testing whether a message can be marked as dead letter when delivery fails or when the message expires.

• The IBM WMQ, Oracle JMS, RabbitMQ, and ACTIVEMQ all support read exceptions, message expiration, and two dead-letter mechanisms.

• ROCKETMQ does not support the message dead-letter mechanism.

• Apollo only supports message expiration mode, which sets the redelivered parameter, which is placed in the dead letter queue when the number of maximumredeliveries (the default is 6) is exceeded.

Basic function--message validity: Mainly by setting the message lifetime at the JMS client, starting the consumer after the end of the lifetime, whether the message is discarded or destroyed.

• The message expiration configuration is supported for products other than ROCKETMQ.

• The RABBIMQ message has a maximum retention period of 2^32-1 milliseconds.

• The ROCKETMQ supports queue expiration settings and deletes the longest-created persistence file when disk space is insufficient.

Basic function-Message priority: The priority of messages can be configured by sending a set of messages to the JMS client, whether they can be processed by the priority of the message.

• Both the IBM WMQ, Oracle JMS, ActiveMQ, and Apollo Support message priority and queue priority configuration.

• RabbitMQ, ROCKETMQ does not support message priority configuration, only supports queue priority configuration, that is, configuring a high priority queue, and a normal priority queue, sending messages of different priority to different queues for processing.

Thematic analysis of operation and maintenance management

• Overall, the products in the operation and maintenance Management module implementation are more complete, there is no obvious short board.

• Commercial products are relatively well-developed in management tools and log systems.

• The function of RABBITMQ monitoring system is more rich.

Operations Management-management tools: in the management interface to test whether the queue can be created, deleted, start and other basic functions.

• Commercial products in the management interface to support the more complete, through the management interface to create, delete and queue start and stop actions.

• Open source products in the management interface only support queue creation, deletion, does not support the queue start and stop actions.

• IBM WMQ, Oracle has a comprehensive configuration file, command line, interface support.

• RabbitMQ, ROCKETMQ management interface does not support the queue start and stop, can be implemented through the command line, interface, support Rest API interface.

• The Actviemq/apollo management interface, in addition to not supporting the start and stop of the queue, does not support centralized management, through the JMX configuration file implementation, support Rest API interface.

Operations Management-Log system: You can modify the log level of the instance, so that it can record important events of the system, such as new inbound connections, new message inbound, and so on.

• Commercial products in the log system to do the most perfect, log configuration is relatively simple.

• Open source products in the message visibility, log segmentation support is not enough.

• RABBITMQ is based on the AMQP protocol and does not support the implementation of third-party log4j log printing.

• Trace log tracing is not supported by ROCKETMQ.

Operation and maintenance Management-monitoring system: can monitor the operation status of message middleware, can monitor the number of queues in the cluster, the number of messages in the queue, the number of messages in and out, and the number of connections.

• RABBITMQ in the Monitoring System module performance is the best, cluster performance, queue performance, the number of messages sent, persistent size, TPS performance, etc. are rich implementation.

• ROCKETMQ does not support the implementation of queue memory consumption and persistence size.

• IBM MQ, Oracle JMS does not support queue memory consumption, persistence size, and implementation of TPS performance.

• Activemq/apollo only supports console implementations of a single queue, but the monitoring implementation of a single queue is rich and does not support the implementation of TPS performance.

Cluster function, transaction support, high availability, stability thematic analysis

• All products except Apollo can realize the functions required by the cluster.

• Apollo does not support cluster functionality (message routing), can be horizontally scaled through configuration files, and can be load balanced through the failover protocol.

• RABBITMQ requires a third-party software haproxy to achieve load balancing.

Cluster function: By configuring the cluster instance, the queue can be expanded horizontally and dynamically, and the message routing and load Balancing can be realized.

• All products except Apollo can realize the functions required by the cluster.

• Apollo does not support cluster functionality (message routing), can be horizontally scaled through configuration files, and can be load balanced through the failover protocol.

• RABBITMQ requires a third-party software haproxy to achieve load balancing.

Transaction support: The application tests whether the commit, rollback, and other mechanisms of local transactions are supported.

• All products are able to implement local transaction support functions, including transaction commit, transaction rollback, and so on.

• RABBITMQ need to purchase a commercial JMS client implementation.

• Rocket uses the custom text protocol to implement transaction commit, rollback and other functions.

High availability: By testing when a queue server fails or a network failure occurs, the client can automatically connect to other queues to keep the business uninterrupted.

• All products can implement failover mechanism, RABBITMQ need to haproxy to achieve failover.

• Only RABBITMQ can implement a non-persisted memory copy of the message.

• IBM WMQ, RabbitMQ, ROCKETMQ, ACTIVEMQ can achieve Master-slave mode high availability.

Stability: Through the continuous sending of messages, the host CPU at more than 60% of the pressure, message middleware running situation.

• Each product shows a good stability, in the continuous 12 hours of high concurrency test situation, can maintain 100% of the message processing accuracy.

Extended feature Thematic analysis

• Each product can be implemented well in file transfer function, event message mechanism and network current limit.

• RABBITMQ file transfer, large file transfer, transaction consistency, and delay queue functionality are required to purchase a JMS client implementation.

• ROCKETMQ does not support large file transfers.

• Apollo does not sustain retransmission, delay queue function.

File transfer function: By testing the file transfer between the server, the statistic data sends and receives the correct rate, checks whether the file is missing or duplicates.

• All products are capable of delivering file transfer functions. The commercial product itself supports file transfer, and open source products need to be implemented through the application.

• The Rabbitmq file transfer feature is required to purchase a JMS client implementation.

Event message mechanism function: through the test, the artificial network disconnection and other unusual events, after the occurrence of the exception event message node will receive the exception event information.

• All products can implement the event message mechanism function.

• IBM WMQ supports the configuration of various events, such as permission events, forbidden events, local events, and so on.

Network current limit function: By adjusting the configuration parameters of the message middleware, testing the allowable network bandwidth, transmitting data and checking the current limit function.

• IBM WMQ is a comprehensive network-limited support that enables network-limited functionality through a variety of configurations, such as message size limits, batch transfer limits, message volume limits, memory usage limits, and disk usage limits.

• Oracle JMS enables network-limited functionality through message size throttling and message volume throttling.

• RABBITMQ supports network throttling by limiting memory usage and disk usage.

• ROCKETMQ supports network throttling through message size restrictions.

• ACTIVEMQ supports network throttling through memory usage limits.

• Apollo supports network throttling through the total message limit.

Continuous retransmission function: Through the artificial disconnection network to test the message middleware's breakpoint continuation ability, statistics data sending and receiving the correct rate, check whether there is data loss or duplication.

• All products except Apollo support the retransmission function.

• Apollo does not support Connectioncontrol command processing and does not enable automatic re-connection, so retransmission is not supported. This feature is expected to be implemented in the Apollo1.8 release.

Large file transfer efficiency: the degree to which large file transfers are supported by a large file transfer test message middleware.

• IBM WMQ, Oracle JMS, ActiveMQ, and Apollo support large file transfers.

• RABBITMQ need to purchase JMS client implementations.

• ROCKETMQ does not support large file transfers.

Transactional consistency: Verify that the message middleware supports final transaction consistency.

• The trade consistency feature is supported for products other than rocket.

• RABBITMQ need to purchase JMS client implementations.

• The open source version of ROCKETMQ does not support distributed things, the commercial version of ONS support.

Delay Queue: Sets the message delay time through the application, whether the message is discarded or destroyed after the delay time, and cannot be processed.

• The delay queue feature is supported in all products except Apollo.

• RABBITMQ need to purchase JMS client implementations.

Overall performance Test Results

• All products in the horizontal expansion of performance testing, have excellent performance.

• Point-to-point performance testing, RabbitMQ, Oracle JMS Performance is fair, ActiveMQ, IBM WMQ, rocketmq slightly insufficient, the Apollo gap is larger.

• Subscription performance testing, activemq a solo show, excellent performance, ROCKETMQ, IBM wmq performance is fair, Oracle JMS, Apollo still have gaps, RABBITMQ gap is larger.

Horizontal scaling Feature Analysis: The two-node, four-node configuration for message size 1KB, persistent, read-write mixed performance test, to obtain the horizontally expanded data.

• RABBITMQ and rocket are significantly superior to other products in terms of their ability to expand horizontally across products. 2-node processing power, RABBIMQ and ROCKETMQ advantages are obvious.

Point-to-point performance Test Special analysis: Through the point-to-point mode, divided into 1K, 2K, 10K, 1M four message size, 8 queues of the cluster for performance testing.

• IBM WMQ performs well in read-write hybrid performance testing, with pure write performance testing not performing optimally.

• Oracle JMS performs well in pure read performance testing, and pure write performance testing does not perform well.

• RABBITMQ supports memory replication patterns in persistence, performs well in pure write performance testing, and is not ideal for pure read performance testing.

• ROCKETMQ does not support non-persistence, pure read performance test performance, pure write performance testing is not ideal.

• ACTIVEMQ performs well in mixed performance testing, pure write performance test performance is good, pure read performance test performance is not ideal.

• The overall performance of the Apollo is poor, only in non-persistent pure write and read-only performance tests.

Subscription performance test Thematic analysis: Through the subscription model, 1K, 2K, 10K, 1M four message size, 8 queues of the cluster for performance testing.

• ACTIVEMQ Overall performance is excellent, whether it is non-persistent or persistent.

• ROCKETMQ does not support non-persistence, and the overall persistence has a good performance.

• IBM Wmq has a balanced overall performance and a slightly better performance test for read-write mixed performance.

• Oracle JMS Overall performance is balanced and performs well in non-persistent read-write hybrid performance testing.

• Apollo Overall performance, read-write mixed performance test performance is good.

• RABBITMQ Overall performance is still a gap, non-persistent performance testing is not ideal.

Open source VS. Business, messaging middleware things you don't know.