A super-thin long-connection message protocol based on Tcp/websockets

This is a creation in Article, where the information may have evolved or changed.

Background

Now when writing the client or webpage, more and more need to deal with the long connection, especially when the boss is going to engage in a chat system, the back-end eldest brother can build a TCP or WebSockets-based message protocol in minutes. But the problem is that with every new project, the back-end bosses can create a new deal, with all sorts of magical restrictions. For example, to maintain a state machine in a long connection, the next message received after sending a message must be XXX, or a complete JSON is directly dropped out and so on. Although all can be used, but need to maintain in various places different bottom communication library, there is no discipline to follow, so drafted this agreement.

Brief introduction

The name STMP of the agreement means that. The project is hosted on GitHub and contains the 最简单的消息协议(The simplest message protocol) complete protocol documentation and related implementations, please visit GitHub at the same time, welcome to submit pr/issue, the address is https://github.com/acrazing/stmp.

Simply put, STMP has the following features:

• Very thin fixed head with only one byte (binary serialization)
• Supports binary serialization (TCP) and text serialization (WebSockets), text serialization supports message subcontracting (passing binary data)
• Upper-level routing control similar to IP protocol masks
• Payload encoding format for protocol transparency
• Heartbeat detection
• Four types of messages: heartbeat, request, notification, reply
• Return status Code control similar to HTTP protocol

The most popular messaging protocol today is MQTT and GRPC, which is defined as A lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks a message protocol that is customized for sensors and mobile devices. The biggest feature is its 固定消息头只有2字节 , as well QoS服务质量控制 . For the former, it is understandable, Any long-connected message protocol should be able to do so, or even simpler (STMP), and secondly its QoS design makes the communication layer more complex, making it more like a Message Queuing protocol than a simple communication protocol. GRPC is based on ProtocolBuffers Developed RPC protocol to achieve. Integration is very high, based on HTTP 2 the bottom, so the versatility is very good, if it is a big project and the team has a certain technical/operational accumulation, is very recommended choice, but this and stmp do not conflict, STMP for the protocol robustness requirements are not high, only need a usable specification of the Enterprise/team, You can use on the web side, or can be used in the client, or smart home and other embedded devices, GRPC, it seems too complex.

Message field definition

In a full-duplex communication system, the two sides need to effectively identify the message sent by each other, and make corresponding processing, choose whether to respond to such operations, so in addition to the actual load, but also need a number of flag fields. In Stmp, the complete list of message fields is as follows, but it is important to note that not every message contains all of these fields, depending on the network environment and the message type to determine the list of fields that should be included. However, if a message contains some of the fields in the following fields, the sort order must be the same in the order in which the fields appear below .

• 消息类型(KIND): Represents the type of message, and the possible values are:
  • 0: Heartbeat message (Ping message)
  • 1: Request message
  • 2: Notification messages (Notify message)
  • 3: Reply to messages (Response message)
• 消息编码格式(ENCODING): Represents the encoding format of the payload, the upper application/codec layer receives the message, the load can be decoded through this field, due to the length of the head limit, the possible range of values 0-7 , the agreed encoding format is as follows:
  • 0: preserves the format, indicating that no payload is present, and that the message must not exist PS and the PAYLOAD field
  • 1: Protocol buffers, reference Protocol buffers
  • 2: JSON, reference JSON
  • 3: Messagepack, reference Messagepack
  • 4: BSON, reference BSON
  • 5: Raw binary data
• 消息ID(ID): The temporary ID of the message, the value range, for the 0x0000-0xFFFF request and reply message, the requester should guarantee that the ID is unique within the time-out period, the reply party with this ID on the reply for the sender to identify
• 消息请求动作(ACTION): Requested action, for upper-level applications for routing control, the value range is 0x00000000-0xFFFFFFFF 32-bit integer, the upper application can be written in xxx.xxx.xxx.xxx a form, similar to IP. The receiving party must be able to correctly identify and transfer to the appropriate processor after receiving the corresponding action. c4/> is a reserved action for use within the protocol. The currently used actions are:
  • 0x00: Version Negotiation (Check Versions)
• 状态码(STATUS): Processing result status code, used in reply message, indicates the result of processing the request, the value range is0x00-0xFF, where0x00-0x7FTo retain values, meaning andACTIONIndependent0x80-0xFFA user-defined status value, meaning that it is based onACTIONDifferent may be different. There are currently defined status codes (similar to HTTP, except for a different value):
  • 0x00 : Ok, movedpermanently
  • 0x10 : +/-301
  • 0x11 : Foun D, 302
  • 0x12 : notmodified, 304
  • 0x20 : badrequest, +
  • 0x21: Unauthorized, 401
  • 0x22 : paymentrequired, 402
  • 0x23 : Forbidden, 403
  • 0x24 : NotFound, 404
  • 0x25 : requesttimeout, 408
  • 0x26 : R Equestentitytoolarge, 413
  • 0x27 : toomanyrequests, 429
  • 0x30 : Internalservererror,
  • 0x31 : notimplemented, 501
  • 0x32 : Badgateway, 502< /li>
  • 0x33 : serviceunavailable, 503
  • 0x34 : gatewaytimeout, 504
  • 0x35 : versionnotsupported, 505
• 负载长度(PS): Represents PAYLOAD the length, in bytes, of the value range, that 0x00000000-0xFFFFFFFF is, the maximum load length 4Gb , whether this field exists or not by the network environment and ENCODING decision, if ENCODING 0 so, or the network environment can be properly subcontracting (such as the websockets environment), This field must not exist, otherwise this field must exist.
• 负载(PAYLOAD): Actual load, length by PS or network subcontracting results determined, encoding method by ENCODING decision, the protocol itself is not responsible for the load codec, need to be explained by the application of the upper layer.

Message type

As mentioned earlier, there are four types of messages in stmp, and different message types may contain fields and meanings that are different, as detailed below:

Heartbeat message

Double-ended in order to ensure the validity of the connection, it is necessary to send a heartbeat message to each other periodically, and this message must not contain any KIND other fields except the other. This message does not require a reply. If a party does not receive a heartbeat message sent by the other person within the agreed time, it indicates that the other party is disconnected or has an exception and should disconnect immediately.

Request message

This message indicates that the sender requested the recipient to return a resource, and if the recipient's reply was not received within the specified time, the wait was discarded and a reply to the upper application was returned STATUS 0x25 , indicating that the request timed out.
This message must contain,,, field, KIND ENCODING ID ACTION may contain PS , field PAYLOAD , and must not contain STATUS a field.

Notification message

This message indicates that the sender sends a notification to the receiving party, and the receiver does not need to reply to this message.

This message must contain, field, KIND ENCODING ACTION may contain PS , field PAYLOAD , must not contain ID , STATUS field.

Reply message

This message indicates that the sender sends a reply message to the receiving Party to reply to a message that was sent to the 请求消息 ID receiver. If the upper-level 请求消息 ID app does not return a message within the specified time, send a STATUS 0x34reply message indicating that the upper-level app processing timed out.

This message must contain,,, field, KIND ENCODING ID STATUS may contain PS , field PAYLOAD , and must not contain ACTION a field.

Serialization of messages

For different network environments, the Protocol has two different serialization methods to deal with, the main reason is that the browser environment in the conversion of strings to ArrayBuffer re-websockets send performance can not be directly looked at (the implementation of the way to refer to Stmp/impl/js/stmp/text.ts, The main is to convert UTF-16 encoding and string into UTF-8 Uint8array), and in order to better web-side debugging, so a set of text serialization scheme.

binary serialization

In binary serialization, the fixed head takes up one byte, contains the KIND ENCODING fields, and if so, it KIND 0 ENOCDING must also be 0 , representing one 心跳消息 . The complete structure is as follows:

|   0 ... 7   |  8 ... 15  |  16 ... 23  |  24 ... 31  || FixedHeader |           ID             |    ACTION   ||               ACTION                   |    STATUS   ||                         PS                           ||                 PAYLOAD    ...                       |

One of the multibyte fields, including,, ID ACTION PS fields, if present, must be BigEndian passed in a way. In addition, the fixed head is as follows:

|   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   ||     KIND      |       ENCODING        |   0   |   0   |   0   |

The last three bits are reserved bits (unused), all 0.

Text is serialized

All fields are concatenated by character | , i.e.:

KIND(1)|ENCODING(1)|ID?(1-5)|ACTION?(1-10)|STATUS?(1-3)|PS?(1-10)|PAYLOAD?(...)

Message segmentation, when using text serialization to pass binary data, the browser environment is not efficient to mix the two together, so it is allowed to be divided into two packets for transmission, the former pass the header information, the latter pass the actual binary PAYLOAD , at this time ENCODING must not 0 be , at the same time, PAYLOAD does not exist in the head package. WebSockets itself guarantees the order of the package.

For one 心跳消息 , there is only one KIND field, so the result must be "0" .

Distinguishing between text messages and binary messages

This is a more interesting place where text messages and binary messages can be completely distinguished by the first byte: for a text message, the first byte is,,, one of the, and '0' '1' for the '2' '3' 0x30-0x33 binary message, either 0x00 (Heartbeat message), Either greater than or equal 0x40 to, because KIND 0 the value must be greater than when it is not 0b01000000 .

Version negotiation

The protocol version has two fields, respectively, MAJOR and MINOR , the value range is to 0 , that 15 is 0x0 , the 0xF form can be serialized MAJOR.MINOR .

The current protocol version is 0.1 .

After the client initiates a successful connection, it needs to send a message to the server with the action, the 0x00 message ID must be 0 , the load Raw is encoded, the payload is the acceptable version number of the client.
List. After receiving this message, the server will reply to a status reply message if it can process one of the list of versions sent by the client, the Ok payload is the selected protocol version
Number, if it cannot be processed, an VersionNotSupported error message is returned, the payload is empty, and the connection is closed.

Serialization of version numbers

In binary messages, a version number is serialized as a 1-byte length of information, where the first 4 bits are MAJOR , and the last 4 bits are MINOR values. Multiple version numbers are connected directly. In a text message, a version number is serialized to 2 bytes of information, where the first 1 bytes are MAJOR , and the next 1 bytes are MINOR values, and multiple version numbers are directly connected.

Realize

At present only realizes the Golang and JS simple message codec part, address in: Go version, JS version, still have a lot of work to do t_t, if someone mention PR just good?????.