MQTT, XMPP, websockets or AMQP? Discussion on the selection good of real-time communication protocols

Wolfram Hempel is the co-founder of Deepstreamio. Deepstreamio is a German-based technology start-up company that provides high-performance, secure, and scalable real-time communication services for mobile clients, as well as IoT devices. Text translated by Jiatong, please specify from the highly available architecture.

Want to send a request to the server and get a response? Use HTTP directly! Very simple. But when you need to communicate through a long-lasting two-way connection, such as WebSockets, you have other options, of course.

This article will briefly explain Mqtt,xmpp,stomp,amqp,wamp and other alternatives. Here are often quoted XKCD comics [1] One:

Wait, there's no such thing as "Real Time protocol"!

That's true, but here I'm using the term (Realtime Protocol) as a McCartney for a bunch of protocols that are designed to distribute messages, synchronize data, and request/respond through a persistent bidirectional connection.

Let's start by classifying them for their respective purposes:

Pure Message

The underlying protocol (for example, TCP) is designed to pass a message from a sender (sender) to a recipient (receiver). They do not matter how the message itself should be built, nor how the message is requested, fetched, stored, and how secure it is.

The protocol above TCP, like WebSockets, adds some additional features, such as using the header to transport metadata, splitting large messages through multiple packets, simple authentication, and routing/redirection related information. Essentially, they are still a point-to-point way of exchanging data.

When it comes to building larger, more complex systems, you need a higher-level communication paradigm:

Publish-Subscribe

Publish-Subscribe mode is a widely used communication method in large-scale systems and is used for many-to-many stateless message delivery. Subscribers (subscribers) can subscribe to the message subject (Topics,channels,events,namespaces), and the publisher (publishers) can publish messages to the message subject, via routing, All Subscribers will receive.

This paradigm is very flexible, efficient and extensible. It isolates the sender from the recipient and allows subscribers to freely subscribe to the topic or unsubscribe. This is the same as our daily subscription to the newspaper.

There are many protocols that support publish-subscribe: Mqtt,stomp,wamp and so on. So how should we choose?

MQTT

MQTT (Message Queue telemerty Transport) [2] is a binary protocol that is primarily used for communication between servers and those with low power consumption of IoT devices (IoT).
It is located on the upper level of the TCP protocol and provides some other features in addition to the basic functionality of publish-subscribe: Different message delivery guarantees (delivery guarantee), "at least once" and "at most once". The re-connected message recovery is achieved by storing the last acknowledged message.

It is very lightweight and suitable for use in unstable network environments, both at the design and implementation level.

when should I use it?

The Internet of Things (IoT) scenario is more appropriate, supports development in almost all languages, and the browser can send and receive MQTT messages through WebSocket.

At the same time, there are many options for Mqtt Broker, such as mosquitto[3] or vernemq[4] and the cloud-based MQTT platform, such as hivemq[5] or cloudmqtt[6].

STOMP

Stream text-oriented message Transfer Protocol (stomp,streaming text oriented Messaging Protocol) [7], is the WebSocket communication standard. Above the usual publish-subscribe semantics, it provides reliable delivery of messages through the begin/publish/commit sequence and the acknowledgement mechanism.

Because the protocol is simple and easy to implement, almost all programming languages have STOMP client implementations. However, there is no advantage in terms of message size and processing speed.

when will it be used?

When used in conjunction with multi-protocol broker (Multi-Protocol broker) middleware such as Apache apollo[8], many interesting integrations can be made. For example, the data for IoT devices is constantly being updated in the browser's diagram.

Select binary or text-based?

So far, we've talked about two protocols: one binary, the other text-based. Let's quickly compare the differences:
Information exchange between computers is achieved by controlling the opening or closing of lights or electricity in the cable (logic switches), or by controlling the peaks or troughs of the WiFi signal. In principle, this is based on the binary form. Therefore, all protocols are binary protocols at this level.
Once the message is sent, the receiver has two choices: it can compose a byte sequence of 0/1 streams to get (parse) the information, or you can perform additional steps, convert it to text, and then parse the text.
The previous method is called (based on) a binary. It saves some expensive operations and is a standard form of transmitting any non-textual information.
For example, images, audio, video or files. Of course it can also be used to send text messages.
For example, by adding several bytes to each message to express meta information, such as describing the length or type of the message, you simply convert the actual message data to text.
However, because the exchange of information is text-based in many published-subscription architectures, many protocol choices simply translate the entire information into text, reducing complexity and improving readability, and of course the cost of performing additional computing tasks after message acceptance.

WAMP

The WEB Application Message Protocol (wamp,web application Messaging Protocol) [9], which attempts to develop an open, text-based protocol standard, combined with a publish-subscribe-based request/response programming model, It also has a powerful routing and message delivery strategy. It is now widely used for the integration of CROSSBAR.IO[10] routers and Autobahn cache clients [11].

Pusher/pubnub&co

Those products of the real-time communications platform as a service (Realtime Platform-as-a-service), such as pusher or PubNub, typically use their own proprietary protocols. Pusher has disclosed detailed specifications [12] that they developed based on the JSON protocol, and encourages third parties or communities to help build clients in different languages. Although Pubnub are more closed, they currently support a range of other open protocols for interaction, such as MQTT.

Queue

In some scenarios, a simple publish-subscribe model is not enough. This is where the queue exists: it supports a variety of different messages and stored patterns.

By acquiring/confirming the policy, the consumer receives some messages from the queue, confirms their "consumption", processes them, and then removes a batch of messages. This is a powerful and common way.

Imagine that you are building an Instagram-like product that allows users to upload images and add various filters. The process of applying filters is a computationally intensive operation. Therefore, it is not possible to operate directly on upload, so the server that receives the image simply records the location in the file system and then adds the task "what image needs to use the filter" to the task queue.

Another machine dedicated to processing filters can get this task from the task queue, read the original image file, apply the filter, and confirm that the task has been consumed (completed). Later, in order to expand the ability of image processing horizontally, you only need to add more consumers (the machine that processes the filters) to process the task queue.

This mode is very easy to extend, and can add complex routing policies, task assignment policies, and so on.

AMQP

The Advanced Message Queuing protocol (amqp,advanced message Queuing Protocol) [13] is a leader in various Message Queuing protocols. RABBITMQ[14] and hornetq[15] are the popular middleware for implementing this Protocol.

when will it be used?

When a simple publish-subscribe model does not meet the usage requirements. AMQP is reliable and powerful. Of course it and its implementation are not lightweight enough. If you need a more lightweight choice, the next thing might be better:

ZeroMQ

ZEROMQ[16] is both a protocol and a set of protocol implementation components. Provides a faster and more centralized alternative to AMQP.

when will it be used?

ZeroMQ is a good choice when you need massive throughput and a message queue without the risk of a single point of failure to support your complex workflow, and of course you need to be prepared for a steep learning curve.

JMS

The Java Messaging Service (Jms,java Messaging service) [17] is a standard implementation of the Java Message Service specification and is part of the Java Enterprise Edition (JEE) specification.

when should I use it?

JMS is the best choice when you use the Java stack and also pay for Java Enterprise Platform. If not, the above-mentioned programmes will be given priority.

Request/Response

Sometimes we just need to send a single request and wait to receive a response, which can be done using HTTP requests entirely. But now that you have established a persistent connection to the server, why not use it?

This communication process for request/response patterns through persistent connections is often referred to as a remote procedure call (Rpc,remote Procedure Calls) or a remote method call (Rmi,remote methods invocation). AMQP or ZeroMQ can implement this pattern through the response queue (Response-queue), and JMS can support Java RMI directly.

Datasync

Datasync is the latest option for real-time communication.

Datasync synchronizes data from the data store to the client. The client changes to the data are synchronized to all subscribers. Datasync hides the details of maintaining data state in real-time communication applications, reduces complexity, and greatly accelerates development, but it is still a non-open protocol standard.

Currently Datasync can be used on several PaaS platforms, such as deepstreamhub[18],firebase[19] or realm[20].

Summarize

Using DEEPSTREAM[21] 's distributed real-time protocol (drp,distributed Realtime Protocol), we have the confidence to find a way to combine all of these concepts into one protocol while ensuring message size, Maximize efficiency in terms of scalability and interoperability.

Reference

1. https://xkcd.com/

2. http://mqtt.org/

3. https://mosquitto.org/

4. https://vernemq.com/

5. http://www.hivemq.com/

6. https://www.cloudmqtt.com/

7. https://stomp.github.io/

8. https://activemq.apache.org/apollo/

9. http://wamp-proto.org/

10. http://crossbar.io/

11. http://crossbar.io/autobahn/

12. Https://pusher.com/docs/pusher_protocol

13. https://www.amqp.org/

14. https://www.rabbitmq.com/

15. http://hornetq.jboss.org/

16. http://zeromq.org/

17. Http://docs.oracle.com/javaee/6/tutorial/doc/bncdq.html

18. https://deepstreamhub.com/

19. https://firebase.google.com/

20. https://realm.io/

21. https://deepstream.io/

Recommended Reading

• What do I care about when designing message middleware?

• How to implement a long-connected messaging system that supports hundreds of millions of of users

• Redis author Another masterpiece: Disque Distributed Memory Queue (i)

• Find a JVM design flaw that causes the GC to slowly grow

Https://mp.weixin.qq.com/s/IyvKMTQY3Nzt719Bs09uyg

MQTT, XMPP, websockets or AMQP? Discussion on the selection good of real-time communication protocols