TCP/IP, WebSocket and MQTT

TCP/IP, WebSocket and MQTT

According to the OSI network layering model, IP is the Network layer protocol, TCP is the Transport Layer protocol, and HTTP and Mqtt are the application layer protocols. Between the three, TCP is the protocol for HTTP and Mqtt at the bottom. Everyone is familiar with HTTP, here is a brief introduction to MQTT. The MQTT (message Queuing Telemetry Transport, Messaging queue telemetry transmission) is an Instant Messaging protocol developed by IBM that is likely to be an important part of the Internet of things. The protocol supports all platforms, almost all networked items and externally connected, and is used as a communication protocol for sensors. The lack of HTTP

The HTTP protocol has been used for many years and has found some deficiencies, mainly in performance, including:

HTTP connection problems, the interaction between the HTTP client and the server is in Request/answer mode, when the client requests, an HTTP connection is established, then the request message is sent, the service side gives the reply message, and then the connection is closed. (later HTTP1.1 support for persistent connections)
Because TCP connections are built up with overhead, the overhead of using SSL/TLS is even greater.

In a browser, a Web page contains many resources, including html,css,javascript, pictures, and so on, so that when you load a Web page, you open multiple connections to the same server at the same time.

HTTP message header problem, now the client will send a large number of HTTP headers, because a Web page may require 50-100 requests, there will be quite a large amount of message header data.

HTTP communication mode problem, HTTP request/reply mode of the session are client-initiated, lack of server notification client mechanism, in the need to notify the scene, such as chat rooms, games, client applications need to constantly poll the server.

And WebSocket is a part of these deficiencies from a different perspective. There are other technologies that are also making improvements in response to these shortcomings. WebSocket
WebSocket provides a mechanism for two-way communication using a TCP connection, including network protocols and APIs to replace Web pages and servers using HTTP polling for two-way communication.

In essence, WebSocket is not limited to the HTTP protocol, but because of the large number of HTTP infrastructure, proxy, filtering, identity authentication, and so on, WebSocket borrow HTTP and HTTPS ports. Because of the port using HTTP, the handshake message after the TCP connection is established is HTTP based, and the server determines whether this is an HTTP protocol or a websocket protocol. WebSocket Connection In addition to the establishment and shutdown of the handshake, data transmission and HTTP is not a little bit related.

For 11 years, WebSocket was finally approved as the IETF's recommended standard: RFC6455. Its predecessor was the work of WHATWG (Web Hypertext Application Technology working Group). and the Web socket API, is the work of the website.

WebSocket can open only one link to the server and exchange information on this link. The advantage is that it reduces the complexity of traditional methods, improves reliability, and reduces the load between browsers and clients. An important reason for this is that many firewalls block ports outside 80, forcing more and more applications to migrate to HTTP.

In the WebSocket draft of 11, some browsers supported older versions of WebSocket, such as the websocket used by Safari on IPhone4 as the old version of the Handshake protocol, and the Handshake protocol used to make the server side. Now only safari supports older versions of the protocol, and the latest Chrome and Firefox editions have been upgraded to HYBI-10 (protocol address). So let's take a look at WebSocket new protocol Hybi-10. The protocol changes are very large, mainly focused on the handshake protocol and data transmission format.

Handshake Protocol

Let's take a look at the general difference: The oldest websocket draft standard has no security key, there are two security keys in draft 7.5 and 7.6, and now there is only one security key in draft 10, which will be in the HTTP header of 7.5, 7.6. Sec-websocket-key1″ and "Sec-websocket-key2″ merge for a" Sec-websocket-key "the value of upgrade in the HTTP header is modified by" WebSocket "in order to" WebSocket " The "-origin" in the HTTP header is modified for "sec-websocket-origin"; Added HTTP Header "Sec-websocket-accept", used to return the original draft 7.5, 7.6 The server returned to the client handshake verification, the original is returned in the form of content, is now placed in the HTTP header, in addition, the server returned to the client's authentication method also changed.

The way the server generates validation varies greatly, let's do an introduction.

Old version:

1 get/http/1.1
2 Upgrade:websocket
3 Connection:upgrade
4 host:127.0.0.1:1337
5 origin:http://127.0.0.1:8000
6 cookie:sessionid=xxxx; Calview=day; daycurrentdate=1314288000000
7 SEC-WEBSOCKET-KEY1:CV ' p1* 42#7 ^9}_ 647 08{
8 Sec-websocket-key2:o8 415 8x37r A8 4
9; " ######

The previous version generates tokens in the following ways:

Take out all the numeric characters in the Sec-websocket-key1 to form a number, here is 1427964708, then divide by the number of spaces in the Key1, get a value, retain the value of the integer digits, get the value N1 ; Take the same algorithm for Sec-websocket-key2, get the second integer N2, connect N1 and N2 in Big-endian character sequence, and then connect with another Key3 to get a primitive sequence ser_key. Key3 refers to the handshake request at the end, there is a 8-byte strange string ";" ###### ", this is Key3. The Ser_key is then MD5 to a 16-byte long Digest, which is the token required for the old version of the protocol, and then the token is appended to the handshake message and sent back to the client to complete the handshake.

New:

1 get/http/1.1
2 Upgrade:websocket
3 Connection:upgrade
4 host:127.0.0.1:1337
5 sec-websocket-origin:http://127.0.0.1:8000
6 sec-websocket-key:erwjbdvalynhvhnulgrw8q==
7 Sec-websocket-version:8
8 cookie:csrftoken=xxxxxx; Sessionid=xxxxx

The new token generation method is as follows:

First the server intercepts the key (length 24), such as 4tajitqo9so2wu8lkrsq3w==, and uses it and a custom string (length of 258EAFA5-E914-47DA-95CA-C5AB0DC85B11) to connect. Then the string is SHA-1 algorithm encryption, get the length of 20 bytes of binary data, and then the data is BASE64 encoded, the end of the service side of the key, that is Ser_key. The handshake succeeded after the server attached Ser_key to the return value sec-websocket-accept.

WebSocket also has its own set of frame protocols. The data message format is as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

0 1 2 3 01234567890123456789012345678901 +-+-+-+-+-------+-+-------------+-------------------------------+ | F| r| r| r|opcode| m|    Payload len| Extended Payload Length | | i| s| s|  s| (4) |     a|             (7) | (16/63) | | n| v| v|       v| |             s|   | (Ifpayload len==126/127) | ||       1|2|3| |             k|                               | | +-+-+-+-+-------+-+-------------+---------------+ | Extended payload length Continued,ifpayload len==127 | +---------------+-------------------------------+ | | Masking-key,ifmask Set To1 | +-------------------------------+-------------------------------+ |          Masking-key (continued) | Payload Data | +-----------------------------------------------+ : Payload Data continued ...: +-------------------------------+ | Payload Data Continued ... | +---------------------------------------------------------------+

FIN:1 bits, used to indicate that this is the last message fragment of a message, and of course the first message fragment may also be the last message fragment;

RSV1, RSV2, RSV3: are 1, respectively, if there is no agreement between the two sides of the custom protocol, then these values must be 0, otherwise must be broken websocket connection;

Opcode:4-bit opcode, define payload data, if an unknown opcode is received, the connection must also be broken, the following is the defined opcode:%x0 represents a continuous message fragment%x1 represents a piece of text message%x2 table not a binary message fragment%x3-7 The opcode that is reserved for future non-control message fragments%x8 indicates that the connection is closed%x9 The ping%xa that represents the heartbeat check, the Pong%xb-f for the heartbeat check, and the retention opcode for the future control message fragment

Mask:1 bits, defines whether the transmitted data is masked, if set to 1, the mask key must be placed in the Masking-key area, the client sends all messages to the server, the value of this bit is 1;

Payload Length: The amount of data transmitted in bytes: 7-bit, 7+16-bit, or 7+64-bit. If the value is 0-125 in bytes, that value represents the length of the transmitted data, and if the value is 126, then the two bytes represent a 16-binary unsigned number that represents the length of the transmitted data, and if the value is 127, The following is a 64-bit non-conforming number represented by 8 bytes, which is used to represent the length of the transmitted data. The number of multibyte lengths is expressed in the order of the network bytes. The length of the load data is the sum of the extended data and the application data, the length of the extended data may be 0, so the length of the payload data is the length of the applied data.

Masking-key:0 or 4 bytes, the data sent by the client to the server is masked by an embedded 32-bit value; The Mask key exists only when the mask bit is set to 1.
Payload Data: (x+y) bit, the load data is the sum of the extended data and the application data length.
Extension Data:x-bit, if there is no special convention between the client and the server, then the length of the extension is always 0, and any extension must specify the length of the extended data, or how the length is computed, and how the correct handshake is determined when the handshake occurs. If there is extended data, the extended data is included within the length of the payload data.
Application Data:y-bit, any application data, after the extended data, the length of the application is = the length of the payload data-the length of the extended data.

Third, MQTT (message Queuing Telemetry Transport, Messages queue telemetry transmission) is a lightweight agent-based publish/Subscribe message Transfer Protocol, design ideas are open, simple, lightweight, easy to implement. These features make it suitable for restricted environments.  For example, but not limited to this: The network is expensive, low bandwidth, unreliable. Runs on embedded devices with limited processor and memory resources.

The protocol features: Use the Publish/Subscribe message pattern to provide a one-to-many message release, decoupling the application.  A message transmission that is masked against the payload content.  Provides network connectivity using TCP/IP. There are three types of message publishing quality of service: "At most once", and message publishing relies entirely on the underlying TCP/IP network. Message loss or repetition occurs.  This level can be used when the environment sensor data, loss of a read record does not matter, because there will be a second time to send.  "At least once" to ensure that the message arrives, but the message duplication may occur. "Only once" to make sure the message arrives once. This level can be used for situations where, in a billing system, duplicate or missing messages can result in incorrect results.  Small transfers, with small overhead (fixed-length headers of 2 bytes), minimize protocol switching to reduce network traffic. Use the last would and Testament features to notify the client of the mechanism of an abnormal interrupt on the parties.

As early as 1999, Dr. Andy Stanford-clark of IBM and arcom company Dr. Arlennipper invented the MQTT (message Queuing Telemetry Transport, Message Queuing telemetry transfer) technology. BM and St. Jude Medical Center developed a set of Merlin systems using MQTT, which uses sensors for home care. St. Jude Medical Center designed a heart device called Merlin@home that can be used to monitor heart attacks that have been implanted with cardioversion-defibrillators and pacemakers (both basic sensors).

The product uses MQTT to pass the patient's immediate update information to a doctor/hospital and then to the hospital for storage. In this case, the patient will not have to go to the hospital to check the heart apparatus, the doctor can check the patient's data at any time, give the suggestion, the patient can check himself at home.

IBM says the transmitter includes a large touchscreen, an embedded keyboard platform, and a Linux operating system.

In the next few years, the application of MQTT will be more and more extensive and deserves attention.

Through the MQTT protocol, dozens of MQTT server-side programs have been expanded to send messages to mqtt through system languages such as php,java,python,c,c#.

In addition, many domestic enterprises have widely used MQTT as an Android mobile phone client and server-side push message protocol. Among the Sohu,cmstop mobile clients, MQTT is used as a message push message. According to Cmstop, a senior research and development engineer who is primarily responsible for message push, Li Wenkei said that with the development of mobile Internet, MQTT, due to the characteristics of open source, low power consumption, will have more contribution in the field of mobile message push, in the IoT field, sensor and server communication, information collection, Mqtt can be considered as one of the scenarios. In the future, MQTT will come into all aspects of our lives.

If you need to download the MQTT server side, you can go directly to the MQTT official website and click Software to download the various versions of the MQTT protocol.

The relationship between Mqtt and TCP, WebSocket can be shown at a glance: