Analysis of technology selection of message queue

2016-02

Message Middleware is a middleware technology composed of message delivery mechanism or Message Queuing pattern, which uses efficient and reliable messaging mechanism for platform-independent data exchange. and the integration of distributed system based on data communication. At present, there are many MQ products in the industry, such as RABBITMQ, ActiveMQ, ZEROMQ and so on are excellent message middleware, but we should choose which is more suitable in the project. This article provides an evaluation comparison of the following Message Queuing products: RabbitMQ, ZeroMQ, ActiveMQ, MSMQ, Redis, Memcacheq

> digression: Here we can think of a little problem. Why a Message Queuing service is required in a Web application. "In a high concurrency environment, due to too late synchronization, requests tend to clog (the main reason), for example, a large number of insert,update and other requests at the same time to MySQL, directly resulting in countless row lock table lock, even the final request will accumulate too much, thereby triggering too many Connections error. By using Message Queuing, we can handle requests asynchronously to relieve the pressure on the system.

RABBITMQ
is an open-source message queue written using Erlang, which natively supports many protocols: Amqp,xmpp, SMTP, STOMP, and so on, makes it very heavyweight and more suitable for enterprise-class development. is an implementation of the AMQP protocol leading implementation of the agent (broker) architecture, which means that messages can be queued on the central node before they are sent to the client. Good support for routing (Routing), load balancing (payload balance), or data persistence. This feature makes RABBITMQ easy to use and deploy, and is suitable for many scenarios such as routing, load balancing, or message persistence, with Message Queuing just a few lines of code. However, this makes it less scalable and slower, because the central node increases latency, and the message is packaged larger. Configuring the RABBITMQ requires the installation of the Erlang environment on the target machine.

? MQ (ZeroMQ)
is known as the fastest Message Queuing system, especially for high-throughput demand scenarios. is a very lightweight messaging system specifically developed for high throughput/low latency scenarios, which can often be found in financial applications. Compared with RABBITMQ, ZEROMQ supports many advanced messaging scenarios, but you must implement individual blocks in the ZEROMQ framework (such as sockets or device).

? MQ (ZeroMQ) is capable of implementing advanced/complex queues that rabbitmq are not good at, but developers need to assemble multiple technical frameworks themselves, and technical complexity is a challenge to the success of this MQ application. ZEROMQ has a unique non-middleware model that you do not need to install and run a messaging server or middleware because your application will play the role of this service. You just need to simply reference the ZEROMQ library, you can use NuGet to install it, and then you can happily send messages between applications. However, ZEROMQ only provides a non-persistent queue, which means that if the down machine, the data will be lost. Among them, the storm of Twitter uses zeromq as the transmission of data streams. ZEROMQ is very flexible, but you have to learn its 80-page manual (if you want to write a distributed system, be sure to read it). The

Zeromq does not have a middleware architecture and does not require any service processes and runs. In fact, your application endpoint plays the role of this service. This makes deployment very simple, but the worry is that you have no place to see if there is a problem. As far as we know, ZEROMQ only provides non-persistent queues. You can implement your own auditing and data recovery capabilities where you need it.

MSMQ
This is the only thing in Microsoft's product that is considered valuable. If MSMQ can prove that it can handle such a task, they will choose to use it. The point is that this thing is not complicated, except for receiving and sending, nothing else; it has some hard limits, such as the maximum message volume is 4MB. However, by connecting with some software such as Masstransit or nservicebus, it can solve these problems completely.

Jafka/kafka
Kafka (the ability to spread messages across different nodes) is a distributed MQ system that was developed and open source by LinkedIn in December 2010 and is now an Apache incubation project, a high performance cross-language distributed Publish/subscribe Message Queuing system, And Jafka was hatched on the Kafka, an upgraded version of Kafka. Has the following characteristics: Fast persistence, can be in the O (1) of the system overhead for message persistence, high throughput, on a common server can achieve 10W/S throughput rate, complete distributed system, Broker, Producer, consumer are native automatically support distributed, Automatic implementation of complex equalization, support Hadoop data parallel loading, for the same as Hadoop, the same log data and offline analysis system, but also require real-time processing constraints, this is a feasible solution. Kafka the parallel loading mechanism of Hadoop to unify the online and offline message processing, which is the focus of the research system of this topic. Apache Kafka is a very lightweight messaging system in contrast to ACTIVEMQ, and is a well-functioning distributed system in addition to its very good performance.

Apache ActiveMQ
ACTIVEMQ resides between the two (RabbitMQ & ZeroMQ), similar to ZEMOMQ, which can be deployed in proxy mode and peer-to-peer mode. Similar to RABBITMQ, it is easy to implement high-level scenarios, and it only requires a low consumption.
ACTIVEMQ is known as the backbone of the Java world. It has a long history and is widely used. It is also cross-platform, providing a natural integrated access point to products that are not Microsoft platforms. However, it can only be considered if it runs past MSMQ. If you need to configure ACTIVEMQ, you need to install the Java environment on the target machine.

One thing to be aware of is that ACTIVEMQ's next-generation products are Apollo,apollo based on ACTIVEMQ prototypes and are a faster, more reliable, and easier to maintain message broker tool. Apache says Apollo is the fastest, strongest stomp (streaming text orientated message Protocol, stream texts-oriented messaging protocol) server.
the features of Apollo are as follows:
Supports STOMP 1.0 and STOMP 1.1 protocols
Topics and queues
Queue Browser
Subject Persistent Subscription
Mirror queue
Reliable Message Delivery
Message Expiration and Exchange
Message Selector
Jaas validation
ACL-based authorization
Support for SSL/TLS, certificate validation
REST Management API

Redis
is a key-value NoSQL database, development and maintenance is very active, although it is a Key-value database storage system, but it natively supports MQ function, so it can be used as a lightweight queue service. For RABBITMQ and Redis on-board and out-of-team operations, each execution 1 million times, every 100,000 times the execution time is recorded. The test data is divided into 128Bytes, 512Bytes, 1K and 10K four different sizes of data. The experiment shows that when the data is compared with the RABBITMQ, the performance of the Redis is higher than that of the 10k,redis when the data is larger than the size of the queue, and the Redis shows very good performance regardless of the data size. And RABBITMQ's performance is much lower than Redis.

Memcacheq
Persistent Message Queuing Memcacheq (abbreviated as MCQ) is a lightweight Message Queuing, Memcacheq feature:
1 Easy to use
2 Faster processing speed
More than 3 queues
4 Concurrency performance is good
5 compatible with the Memcache protocol. This means that as long as the memcache extension is available, no additional plugins are required.
6 It is also very convenient to use in the Zend Framework.

in the end, these several products:
1. Each has its own client API or supports multiple programming languages;
2. There are a lot of documents;
3. Positive support has been provided.
4. ActiveMQ, RabbitMQ, MSMQ, Redis all need to start the service process, which can be monitored and configured, and a few others have problems.
5. Relatively good reliability (consistency), scalability and load balancing, and of course performance

There is no more nonsense here, the following is a set of test results intercepted from the Internet. Displays the number of messages per second sent and received. A total of 1 million 1K messages were generated throughout the process. The execution of the test is performed on a single Windows Vista stand-alone.

As you can see, ZEROMQ and others are not a level. Its performance is surprisingly high. Despite this, this product does not provide message persistence, the inability to store and monitor intermediate processes, the need to implement audits and data recovery, and therefore is not satisfactory in usability and ha. The conclusion is clear: if you want an application to send messages as soon as possible, you choose Zeromq. It is more valuable when you are less concerned about accidental loss of certain messages.

The future of this blog is more likely (and not very desirable) to choose to use the RABBIT,RABBITMQ built-in Ha, if you build cluster, load balancing and other issues there is no need to worry about the same time you can set up a queue mirroring. But this kind of thing should be done more testing, you will eventually have a favorite, I hear, read all kinds of things about rabbit I think it should be the best choice.
> Recommended reading : "Performance test, do you really understand?"

A list of other queues, which are not analyzed here, if you need to know more about Google yourself.
If You can do a better test result-just comment It is here, could update this post. Thanks.