Architecture Design: Inter-system Communication (--MQ): Message Protocol (UP)

1. Overview

Starting from this article, we introduce another type of inter-system communication and transmission: MQ Message Queuing. First, we will discuss the basic principles and working methods of several common Message Queuing protocols, including MQTT, XMPP, Stomp, AMQP, Openwire, and so on. Then, on this basis, we introduce two MQ products: Activemq and RABBITMQ, which are now widely used Message Queuing software in business systems. Includes their installation, operation, support protocols, clustering, and invocation methods.

Of course, in this process we will also mention other Message Queuing protocols (or implementations), such as Microsoft JBOSSMQ, MSMQ, commercial product WebSphere MQ, Oracle Advanced Queue (AQ), and so on. We'll also discuss the relationships between these dazzling protocols, software, and libraries .

Then we'll take some time to discuss the emerging message queue Kafka and ZEROMQ. They are used more and more widely, especially in the collection of big data. Finally, we will use Message Queuing to build a high-performance log collection system as a combat.

2. Basic concept 2-1, message

First there are three basic concepts that we need to discuss before we start: message, message protocol, Message Queuing. The message is both a carrier of information and a description that the reader can understand. in order for both the message sender and the message receiver to understand the information that is hosted by the message (the message sender needs to know how to construct the message, and the message recipient needs to know how to parse the message), they need to describe the message in a uniform format called the Message Protocol . Therefore, a valid message must have a format, and a message without formatting is meaningless .

There are two ways in which messages can be sent from sender to receiver. One is what we can call Instant Messaging , which means that when a message is sent from one end (the sender of the message) can immediately reach the other end (the message receiver), the specific implementation of which is the RPC we have already described (and of course, the simple HTTP communication satisfies this definition) Another way is called deferred message communication , that is, after a message is emitted from one end, it enters a container for temporary storage, and when a certain condition is reached, it is sent to the other end by this container. a specific implementation of this container is Message Queuing .

2-2. Knowledge structure

Message Queuing is the same as the RPC that has been introduced: either RPC or Message Queuing they are built on the network IO model (we've covered a variety of network IO models). The Advanced network IO model will give the MQ protocol excellent performance (and, of course, performance depends not only on the network IO model).

As you can see, the implementation of a message communication software (or library) is based on the "Protocol" : The RMI Library is built on the RMI Protocol (the RMI protocol is part of the Java Specification Protocol) and belongs to an "instant message communication" The RABBITMQ and qpid messaging software is designed based on the AMQP protocol, which belongs to a "deferred message communication".

Although the message protocol has a "private protocol" and an "open protocol" (whether to open the message specification document to the industry or whether to allow an organization to change the agreement), a software (library) does not necessarily support only one protocol (for example, ACTIVEMQ implements a variety of message protocols). Although a protocol may not necessarily have only one software (library) implementation (such as a library that supports the WebService protocol Codehaus XFire, Apache CXF, Jboss resteasy, etc.), this does not affect the " The realization of a certain kind of message communication software (or program library) is based on the concept of "protocol", but the basic concept is strengthened.

3. Message Protocol

So to understand Message Queuing, we should start with these message protocols that support "delay message communication." In this section we will first introduce several messaging protocols used by the readers, XMPP, stomp, and AMQP. In order to take the MQ software we explained later, we also focus on the AMQP protocol in these three Protocols.

3-1. XMPP protocol 3-1-1, definition

XMPP is the extensible Messaging and Presence Protocol, a set of open technologies for instant Messaging, presence, multi- Party chat, voice and video calls, collaboration, lightweight middleware, content syndication, and generalized routing of XML data.

The above content is quoted from the XMPP official website, which clearly indicates the purpose and characteristics of the XMPP protocol. XMPP's predecessor was Jabber, an open source form of organization-developed network Instant Messaging protocol. XMPP is currently being standardized by the IETF International Standards Organization.

XMPP is XML-based for the development of IM systems. The domestic more popular XMPP server is called OpenFire, it uses Mina as the lower layer of the network IO Framework (not MINA2 is MINA1); foreign use of more than the XMPP server is called Tigase, its official website claims that a single node can support 500,000 of users online, Cluster can support 1 million users online: (http://projects.tigase.org/)

Cluster with over 1mn online users. 500k online users on a single machine

Of course, if the reader's company needs to develop an IM system, in addition to using out-of-the-box XMPP servers, the client or development package for the XMPP protocol (for extended development) needs to be implemented. You can view the official XMPP recommended development package on the XMPP website, and support for all languages is basically: http://xmpp.org/software/libraries.html

I have been involved in the development of a number of IM systems (including their own entrepreneurial projects), in general, the XMPP protocol itself is a good choice, but it will take a considerable amount of time to learn, and some XMPP client, server-side or the library does not have these development team propaganda so stable and useful. so if your company needs to develop an IM system, then creating a proprietary messaging protocol can be a good choice .

3-1-2, protocol Communication Process Example

To allow readers to have a perceptual understanding of the XMPP protocol, here we give an XMPP protocol to handle the "IM user login" operation process (XMPP login method is divided into the user password and no user password two ways, here we introduce the method of no password login).

The XMPP protocol itself is rich in detail, and here we only discuss login operations, if the reader is interested in downloading the full set of XMPP official specification documents for Research (http://xmpp.org/):

You can see the XML fragment in the XMPP protocol. There are several key messages in the XMPP protocol, such as:

• Stream tag: A traffic token that refers to a communication request (or response) initiated by the client or server side of XMPP. Traffic does not carry positive content information indicating that an interaction has occurred between the client and the server side. The properties of stream include: To, from, ID, Xml:lang, version, and so on.

• IQ Mark: IQ Mark is info/query abbreviation (you can understand as query Information request), generally is a set of form, by the client initiates the query request, by the server side returns the query result. Because the type of query request is different, there are many child tags that can be embedded in the IQ tag. For example, you can embed the bind tag to indicate a binding relationship between a user and a Jid, and you can embed multiple item tags that indicate the user's friend information (as follows) that the query was given.

<iq to =' [email protected]/someresource ' type=' result ' ID =' roster '>      <query xmlns=' Jabber:iq:roster '>          <item jid=' [email protected] ' name=' someone1 '/>          <item jid=' [email protected] ' name=' someone2 '/>      </Query>  </iq>

• Jid tag: Jid (Jabberid) is an XMPP protocol that identifies each XMPP entity in an XMPP network (an entity can be a user, a server, a chat room), and the canonical format is as follows:

jid = [ node "@" ] domain [ "/" resource ] 

• There is also a message, presence tag: message is an entity content tag that records the real content of the chat, and the presence tag indicates the service status of the XMPP user (offline, online, busy, etc.). Examples are as follows:

<message to="[email protected]/someresource" type="chat">     <body>helloword。。。</body></message>

3-2, stomp agreement 3-2-1, definition

Stomp protocol, English full name streaming text orientated message Protocol, Chinese name is ' Flow text oriented messaging protocol '. is a plain text-based protocol (text-based means that its message format specification does not have the same XML format requirements as the XMPP protocol, you can think of it as ' semi-structured data '). There are currently two versions of the STOMP Protocol: V1.1 and V1.2.

A standard stomp protocol consists of the following sections: command/Information keywords, header information, text content. As shown in the following:

Here is an example of a simple protocol message:

CONNECTaccept-version:1.2someparam1:value1someparam2:value2thisisconntecon ^@

In the example above, we used the Connect command of the STOMP protocol, which meant to connect to the STOMP proxy and carried the version information and two custom k-v information that required the proxy side (note the ' ^@ ' notation, which is used in the STOMP protocol to represent null).

There are two important roles in the STOMP protocol: The STOMP client and any stomp message broker (broker). As shown in the following:

After reading the above, some readers may ask: why is it called the Stomp message agent, not the Stomp messaging service? Because Stomp Broker is only responsible for accepting and storing messages sent by clients, only forwarding messages as required by the client, just managing client connections and Subscriptions: it is not responsible for any business processing based on the content of the message. So it is more appropriate to call it the message broker side.

since the structure of the STOMP protocol is so simple that any technician who understands the STOMP protocol command format can develop STOMP proxy or stomp client, The system that satisfies the stomp protocol is easily accessible to another third-party system (such as ACTIVEMQ) that also satisfies the STOMP protocol.

3-2-2, basic command/return information

Similar to the way the XMPP protocol is introduced, in order to allow readers to further understand the STOMP protocol, this section introduces the basic commands of the STOMP protocol and the kind of information returned by the proxy, and enumerates some examples for use.

In the STOMP protocol, there are mainly the following command/return information (some articles also called a complete information frame). These command/return information forms the main body of the STOMP protocol and enables your stomp client and stomp Agent to complete the process of connection, send, subscribe, transaction, and response. These commands/returns are:

• connect/stomp Command : The client connects to the Stomp proxy side by using the Connect command. If you use the Stomp command, the version of the Stomp proxy must be 1.2.

• Connected Information : When the Stomp Agent receives the Connect command sent by the client and the processing succeeds, the connected status information is returned to the client, and the error message may be returned if there is any problem in the process

• Send command : The client uses the Send command to send content to a specified location (a virtual path on the proxy side). This will enable other clients that have subscribed to the message event on this path to receive this message.

• SUBSCRIBE Subscription Command : The client uses the SUBSCRIBE Subscription command to subscribe to the Stomp Service proxy for listening on a virtual path. This allows the client to receive this message when other clients send content to this path using the Send command. When using subscribe, there is an important ACK attribute. This ACK property indicates whether the message sent to this client by the Stomp Service proxy requires an ACK command before the message is considered successful. As shown below:

SUBSCRIBEid:XXXXXXXXXdestination:/testack:client^@

The Subscribe command information above indicates that the virtual location of the client subscription is test. and the ACK attribute in the command information is client, stating that when the client receives the message, the ACK command must be sent to the agent, and the agent will assume that the message has been successfully processed (there are only three ACK values: Auto (default), client, and Client-individual).

• Unsubscribe Unsubscribe Command : The client uses this command to suppress the interception of message events on a path. If the client has not subscribed to this client before the path is given, the command execution is invalid.

• Message message: When a client receives a message at the location of a subscription, the message is described by the Word keyword. Similar to the following information is the description of the message received from the agent:

MESSAGEredelivered:truemessage-id:ID:localhost-34450-1457321490460-4:24:-1:1:1destination:/testtimestamp:1457331607873expires:0priority:42345431457331607861

• Begin Transaction Command : The STOMP protocol supports transaction mode in which messages sent from a client using the Send command are not actually sent to the stomp proxy until the commit is formally committed. The begin command is used to open a transaction. Note that you can have one message in a transaction, or you can have more than one message .

• Commit commit command : Use this command to commit a transaction when the information definition in the transaction is completed. only one or more messages in one transaction will enter the queue on the Stomp proxy (the other clients subscribed to the event to receive these messages) only after the commit command is used.

• Abort Cancel/Terminate a transaction command : It is clear that this command is used to cancel/terminate a transaction that does not currently have a commit command executed.

• ack Acknowledgement command : When the client uses the Subscribe command to subscribe, if the ACK attribute is set to client in the Subscribe command, then the client receives a message (ID xxxx). The ACK command must be sent to the Stomp agent, so that the agent will consider the message processing to be successful, and if the Stomp client does not send an ACK command before disconnecting, the STOMP proxy will send the message to the other client after the client disconnects .

ACKid:MESSAGEID^@

Note the id attribute of the Head section, and the id attribute passed is the ID of the message message that you received earlier.

• NACK does not confirm the command : The same is the state of the above subscribe command, if at this time the Stomp client sends NACK information to the STOMP proxy to prove that the message failed in this client processing. The STOMP proxy will send this message to another client (regardless of whether the current client is disconnected).

• DISCONNECT Disconnect Command : This command disconnects the STOMP client from the Stomp proxy side.

              （接下文）

Architecture Design: Inter-system Communication (--MQ): Message Protocol (UP)