Message design for interprocess communication using MQTT

1. Why inter-process communication is required

The realization of complex business system has two kinds of relative realization idea, one is to do a chatty program, this program is all tube, completely realizes all functions of the business system, second, the complex system is decomposed into a series of function points, then corresponding to do multiple programs to jointly realize the overall function. The latter idea is the embodiment of UNIX philosophy:

The program should focus on only one goal and do it as well as possible. Allow the program to work together with each other. The program should be allowed to handle the text stream because it is a common interface.

So, this is the context of interprocess communication. A number of independently running programs, in order to collaborate to complete the overall function of the system, it is necessary to do data interaction, also known as inter-process communication.

2. Features of the MQTT message 2.1. Message notification based on the topic "Subscribe, publish"

This is the greatest feature of the delivery of MQTT messages. One of the great benefits of this is that the sender and receiver of the message are decoupled, and each follows a pre-defined theme and message assembly format. The topic-based subscription and publishing mechanism is still easy to handle, especially in complex scenarios where the message is multi-producer and multi-consumer.

2.2. Text-based data flow

Message publishing requires two elements, one topic (subject), and the other is payload (payload). Topic is a custom string that is the text data stream mentioned in the UNIX philosophy. This is very versatile and very flexible. For example, it is common to distinguish multiple levels in a topic by slash/. The message design in the next section can be seen.

Then the data in the payload, we can also be organized into a text data stream, that is, the use of very lightweight and common JSON format.

2.3. Other Features

Other features such as message encapsulation overhead, transport mode support for multiple QoS, etc., we are not particularly concerned here, not in detail.

3, three different angles of message design

The message design for interprocess communication based on MQTT, as described in section 2.2, contains both topic and payload designs.

The message that needs to be passed payload in addition to the large amount of data used to share memory, and so on, other basic can use JSON text to organize delivery. In different business systems, the content of the message varies widely, so the structure design of the JSON has a chance to explain it later.

The following main talk about topic design. Do not underestimate this small section of the topic string design, different angle design will affect the design of the entire system structure. The "angle" here is three: based on the angle of the message data, based on the angle of the message receiver, and the angle based on the sender of the message.

For ease of interpretation, let's start by setting up a system with the following processes, data, and their relationships:

Process/Data	data1	data2
Prog_a	Producer	Producer
Prog_b	Consumer	Producer
Prog_c	Consumer	Consumer

Note: Producer is the process that produces data that needs to be sent to other processes, and consumer only receives data from other processes.

3.1. Theme design based on message data

Based on the message data, the literal meaning of the topic text string reflects exactly what data is carried in the payload.

We design the MQTT topic and payload JSON format in a system, first of all processes need to transfer the data to do a unified comb, to classify, and distinguish different levels. You can then indicate the nature of the data carried by the topic by setting the corresponding field in each topic. Glance.

For the processes and data in the table above, we will design two topic from the data, namely:

/data1

/data2

The consumer process that needs to receive the data is responsible for publishing the subscription to the corresponding Topic,producer process.

For example, the data transmitted in one of our projects actually includes configuration information, data acquisition, and data collected in real-time data collection and analysis. Then we design a topic to transmit real-time data acquisition, it can be designed as/DATA/REALTIME/1. The last 1 represents Channel One, which distinguishes it from the other channels, and the preceding data is distinguished from the configured topic Config. So here topic read the literal meaning is to describe the data itself, do not care about the sender is how, do not care who is the recipient.

3.2. Theme design based on message recipients

Facing the same process and data in the above table, from the message receiver point of view, the first is Prog_b receive Data1, the second is Prog_c receive Data1, the third is Prog_c receive data2. The corresponding three topic are:

/prog_b/data1

/prog_c/data1

/prog_c/data2

This way, watching topic will know who will receive the message and what the content is.

Of course, the received data content will not be represented in the topic as in the 3.1 angle. Further, the latter two topic can be merged into one, since both are the same recipient, data1 and data2 can be distinguished in the JSON data organization of payload. The simple thing is to add a message ID field to the JSON. This can be implemented at the time of all the data received processing in a topic processing function is processed separately.

3.3, based on the message sender's theme design

Designed from the message sender's point of view, there are three topic:

/prog_a/data1

/prog_a/data2

/prog_b/data2

This design is relatively close to data-based design. Based on the description of the data, he shows who sent it. In general, data producers are well aware of what data they produce, but are not sure who cares about the data. Then he just made the production, but released it.

Comparison of 3.4, three design methods

The first is the most authentic and topic design perspective in the MQTT framework that best fits the original design of its framework. High level of abstraction, good scalability. First consider what the system has data, design the corresponding topic, do not care who will produce the data, who will consume the data. The design implementation of producers and consumers can be changed at random, without affecting the previous design of topic. The disadvantage is that the level of abstraction is high, the implementation of the code will become a bit "impossible", the whole system is not so smooth to understand.

In contrast, the second design from the point of view of the message receiver is the highest coupling in three kinds. A topic, who would subscribe to him, and for the data producer, when releasing the data he produced, to know who was concerned about the data and to publish the corresponding topic. This design can be used when the system complexity is low, the coupling degree is high, but more specific, when the implementation of topic design and coding is closely linked. Easy to understand.

The third design, based on the sender's perspective, is between the two before as a compromise. Should try more.

Rio Jeliang-2018-1-27

Message design for interprocess communication using MQTT