Non-serious rethinking of the MQTT 3.1 protocol

Objective

The MQTT 3.1 protocol is not good enough in a weak network environment (such as 2G/3G, etc.) and is therefore reflective.

Performance under weak network environment

Mobile phones and other terminals in a weak network environment packet loss will be very obvious, the connection to MQTT server success rate is very low. Compared to simple requests-the corresponding model of HTTP, its success rate is much higher than the success of an MQTT subscription.

After each TCP disconnection or disconnect, the terminal will immediately initiate TCP reconnection, the connection is successful, the previous step is repeated to send the connection command (connect), the subscription command (subscrible), indicating that there is no problem with these procedures, The problem, however, is that the connection from the terminal to the server is successful, and to the sending of subscription commands, the process becomes expensive in the case of a weak network:

从TCP建立开始的三次握手到完整的订阅命令发送完毕，考虑到TCP堆栈的每次接收数据方响应ACK，这中间终端和服务器端至少产生了10次数据交互。

In a 2G/3G network environment where network changes are frequent or less stable, this process can be tedious and inappropriate, and will add to the burden of the already weak network load.

Under the weak network, in any phase of the execution process, there is the possibility of a sudden network interruption problem:

1. The TCP link could not be successfully established or died during the three handshake, or the packet was lost after the handshake, or the client connection timed out too small

2. After the connection is established, the Connect command is sent, or the TCP ACK acknowledgement packet is not received, or the client waits too low, causing the subscription command to fail to interact

3. The Send Subscrible command failed after sending the subscrible command, but the server side was confiscated, or because the packet was dropped or the network was disconnected

4. The subscription failed when the subscrible command was successfully sent, or the mobile network was disconnected (some operators have timed out for requests that consider HTTP), or wait for a timeout

TCP is a non-aware virtual connection, the middle of the two ends will not be immediately notified, otherwise there is no need for a heartbeat keepalive mechanism.

For example, a server on the wire, based on log analysis, receives only the connection command (connect) but no subsequent subscription commands (subscrible), with millions of levels per day.

In short, MQTT does not perform well for low-rate, weak network environments.

Improvement Point

Business Improvement points:

1. Client connection Timeout, wait timeout set larger, two seconds too short, can be set longer, such as 10 seconds
2. Server-side support after receiving the user sends the CONNECT command, instantly sends some live data/hot data (long-cached), similar to the HTTP request-the corresponding model, the purpose, some hot data sent to the terminal to be early, the faster the better (so famous to be early) This requires both client and server side support.

Protocol Improvement points:

1. Connect command variable header contains "MQISDP" too much, college style.
2. Allows the subscription topic string to be carried in the load (payload) in the connection command
3. Allows the topic of the last connected subscription to be changed in the connection command. No, carry subscription business, although redundant, but practical. Eg: subscription to the TOPIC has not changed, TOPICchange:0; unsubscribe, unsubscribe:topicone;subscribe:topic_two
4. PUBLISH, Puback and other supported Message Identifier only 16 bits, too short, the actual business can not be globally unique. Introduce the mapping correspondence between mid and business ID? That is the state, need maintenance, the price is quite high. Industry popular View, the state-of-the-architecture is convenient for horizontal, vertical, longitudinal, four-direction expansion, hehe. The best way is to modify it to support string form, otherwise the maintenance cost is high!
5. Heartbeat command pingreq/pingreq can do a byte transfer, save a byte, some obsessive feeling
6. Low-rate networks need to do some compatibility and tuning

Some suggestions seemingly redundant, batch or packaging processing is more efficient than a single processing, more resource-saving, weak network environment requires the interaction to be as small as possible, the data is to be instantaneous arrival, the faster the better.

Rigorous tiering and business decoupling can lead to performance issues. Like the current Linux kernel TCP/IP network stack hierarchy is clear, each layer has its own functions, but and directly bypass the kernel directly run in the user state (Users Space) packet I/O, processing performance is not in a grade, such as Netmap, DPDK and so on.

Mqtt-sn

MQTT is not available for terminal environments that do not support network stacks such as TCP/IP.

In some similar to the sensor electronics, the resource is very limited, insufficient computing power, embedded in the TCP/IP network stack is not realistic, a better way based on the IEEE 802.15.4 for low-speed wireless personal domain network (lr-wpan) of the physical layer and the media Access Control layer specification on the transmission of UDP packets, Each packet has a maximum of 128 bytes.

The MQTT-SN (MQTT for sensor Networks) protocol is designed to be very limited to similar sensors, and the protocol process architecture is more interesting:

More details of the agreement, to be read further.

TCP is not the most suitable mobile network transport protocol

Let's count the number of bytes transmitted over the network.

Ethernet frame head At least 18 bytes, ip header fixed 20 bytes, TCP header 20 bytes (UDP head 8 bytes), plus Telecom broadband billed PPPoE 8 bytes:

• TCP packet header information occupies at least 66 bytes
• UDP Datagram Header information occupies at least 54 bytes

UDP can save 12 bytes over TCP.

The MQTT-SN protocol chooses to use UDP to see its efforts in saving resources.

Then look at the weak network environment.

• In the case of network availability, UDP can send data to the destination during TCP's first handshake
• During the three handshake, the UDP client and UDP server can complete a complete interaction at the data plane (ping-pong)

In the case of bad network, the timeliness of UDP is better than that of tcp,tcp long connection, so that the process of establishing complete interaction is very low success rate. The result of the low latency and real-time performance of UDP in this case is very prominent.

TCP or HTTP is theoretically a reliable connection, but not as reliable when the network is bad. After the client generally submits the HTTP request, there is no confirmation that the commit is successful, and in the weak network environment, the packet is dropped and the server is not receiving it. Another TCP network stack will have the packet re-send mechanism + application layer re-send request, may cause the kernel processing multiple packets of re-send (and congestion window will shrink, the packet speed slows), may aggravate the load of weak network.

Compared with TCP, UDP is not connected, it represents fast, resource consumption is small, the outstanding performance is small delay. As for packet loss without retransmission, the upper layer of the Business-level application protocol/mechanism can ensure that lost packets are re-sent or reissued, and will be more transparent, secure control rights. While TCP packets are re-sent, upper-level applications have no control rights.

Connection protocol Aspects:

• TCP-oriented connectivity generates state management and maintenance, with a small cost, such as a client reset exception that is often seen. A complete request cycle must be fixed on a single server
• UDP is not a connected feature. Each request packet can be randomly assigned to a different machine for processing and can be completely stateless scale-out

In short, for real-time special functions, or fast access to the characteristics of the terminal, may wish to consider UDP. However, many times UDP and TCP people will be mixed use, will compensate each other for their shortcomings.

Summary

If the MQTT protocol does not meet business requirements, you may want to consider customizing, simplifying the process, or using UDP to re-implement, or using tcp/http as a supplement.

Non-serious rethinking of the MQTT 3.1 protocol