Message Queuing technology point carding (Mind Guide chart)

Message Queuing technology point carding (Mind Guide chart)

As the communication middleware between service/application, Message queue can play the role of business coupling, broadcasting message, guaranteeing final consistency and fault peak flow control (overcoming the bottleneck of short plate). This article does not intend to explain in detail the message queue, but the systematic carding message queue may involve the technical point, plays the role of sketchy, constructs a macroscopic concept, uses the mind map to comb.

Before the introduction, briefly compare RPC and Message Queuing. RPC is mostly in request-response mode, and also includes an increasing number of responsive paradigms, which are superior to Message Queuing for communication between service/applications that require point-to-point interaction, strong transaction assurance, and latency-sensitive services. Then the message Queue (hereinafter also referred to as MQ, message Queueu) can be seen as an asynchronous RPC, which changes the RPC to two times, dumps the content, and then delivers it at the right time. Message Queuing middleware is often a distributed system, and the communication between internal components still uses RPC.

At present, the open source industry with more than the selection includes, ActiveMQ, RabbitMQ, Kafka, Alibaba notify, Metaq, ROCKETMQ. The technical points below are also used to learn from these open source components and then extract some common technical points.

For a systematic understanding of Message Queuing, see the mind map below.

1. Overall architecture

Generally divided into three producer,broker,consumer.

2. RPC Communication

Refer to the "System-aware RPC" (HTTP://WWW.INFOQ.COM/CN/ARTICLES/GET-TO-KNOW-RPC) in detail.

3. High Performance Assurance

The main consideration is MQ latency and throughput.

High performance delivery, divided into producer and broker considerations. Producer can synchronize mutation step, single variable batch to ensure high performance of the transmitter, batch send trigger conditions can be divided into buffer full or time window to the. Broker can be divided into multiple topic, and then multi-partition/queue to divide and conquer (Divide and conquer) strategy, increase the degree of parallelism, decentralized delivery pressure. In addition, broker for the need to persist the message, can use the sequential Io,page cache, asynchronous brush disk and other technologies to improve performance, but the asynchronous brush disk in the case of power loss, may be lost data, can be combined with the following high-availability scenarios, the data is strictly not lost and high performance throughput between the compromise.

High-performance consumption, that is, consumer and broker communication, to push and pull messages. Using consumer group to scale consumption capacity, you need to use partitioned or out-of-order consumption in accordance with business scenarios. The 0 copy technology saves broker-side user-to-kernel data copies, which are sent directly from the page cache to the network, maximizing delivery performance. Consumer batch Pull,broker batch push. Broker side can also do message filtering, can be implemented via tag or plug-in.

4. High Availability Guarantee

primarily for broker.

Cluster high availability, producer through broker delivery message, so there must be and only one broker Master is responsible for "write", select the main policy is divided into automatic selection and non-active selection, automatic selection of the main use of distribution consistency components, such as Kafka use zookeeper, non-automatic selection of the main, For example, ROCKETMQ relies on multiple stateless name servers.

Data is highly available for the broker persistence backlog message scenario. Can be done with the help of distributed storage, but often performance is a short board, so most mainstream products are local IO sequential write, master-slave backup, multi-replica copy to ensure availability, such as ROCKETMQ divided into synchronous double write and asynchronous replication, the former like HDFs, After writing multiple copies and then returning to producer success, there is a certain performance penalty, but not much, the latter maximizes performance, but when the primary hangs, the data is at risk of loss.

Similarly, MQ clusters need to consider high availability across rooms (not "offsite"), Broker writes are highly available, consider minimizing mttr, while not blocking consumer consumption.

5. Scalability Guarantee

The Divide and conquer strategy is used to increase the degree of parallelism of delivery and consumption, multiple topic, multiple partitions/queue, multiple replicas, multiple slave, or mirrors.

6. Agreement

Protocols for producer, consumer, and broker communications, including AMQP, STOMP, MQTT, HTTP, Openwire (ActiveMQ), XMPP, custom, and so on.

AMQP is a more comprehensive and complex protocol, including the Protocol itself and the model (broker, Exchange, routing key, etc.), currently RABBITMQ is the most famous open source implementation of AMQP Message Queuing, There are very many languages that support communication with Message Queuing based on the AMQP protocol and can support protocol access such as Stomp, MQTT, etc. via plugins. Both Kafka and ROCKETMQ use custom protocols.

7. Consumer Relations

Includes three types of

1) Point-to-point, which is the P2p,fifo queue, can be considered unicast.

2) topic mode, Pub/sub Publish subscription.

3) fanout broadcast mode.

8. Message stacking capability

For persistent messages, if stored on a local disk, you can use both the Sync Brush disk and the asynchronous brush disk strategy. The disk does not accumulate indefinitely, there will be a cleanup strategy, such as Kafka, ROCKETMQ are based on time, the amount of data retention.

Non-persistent, only in memory, consumers can choose to delete the finished processing.

9. Reliable Delivery

For producer, synchronous, asynchronous, from API and I/O levels can be used, single, batch for the throughput level. Fire-and-forget mode, similar to UDP, although it can be sent. For possible errors, such as connection Broker failures, RPC timeouts, post message failures, post-publish no response, optionally ignore or re-send, so often repeated delivery is unavoidable.

For broker, if you want to ensure that data 100% is not lost, it is possible, but need to sacrifice performance and throughput, the use of synchronous multi-write, multi-copy policy + Synchronous brush Disk Persistent message can be strictly guaranteed not lost. In addition, the broker also makes integrity checks for payload that write messages, such as CRC.

10. Reliable Consumption

The number of consumption, including at the most once, at least once, exactly once, of which the first two better to do, the final exactly once need streaming consumer system and broker-side collaboration to complete, such as Storm's Trident and Flink.

Push or Pull mode. Push mode minimizes delivery delay, but does not consider the consumer load capacity, pull is generally polled to receive the broker's data, according to consumer own ability to consume.

Consumption record point, typically each message has an offset, id, or timestamp, consumer can follow this offset for fixed-point consumption and message replay.

The message confirms that the consumer consumption completes the ACK callback broker or the cluster high-availability middleware (ZK) notifies the consumption progress.

Error handling, in the case of consumption failure, you can reply to Nack, request to re-send the/requeue message, when the error exceeded a certain threshold, put into the dead letter queue.

Repeated consumption of messages, which is related to the number of consumption, consumer in some cases need to do idempotent, to ensure that repeated consumption will not cause business anomalies.

11. Message Type

Sequential messages, ordered words, divided into partitioned orderly or global order, the former can be modeled according to a business ID, sent to different partitions/queue can, the latter often need a single queue to be satisfied. Disorderly consumption maximizes throughput.

Timed messages, transactional messages, such as ROCKETMQ, are supported.

12. Message Query