The essence of Message Queuing design is basically hidden in this article.

Today there are many mainstream information middleware, such as veteran activemq, RabbitMQ, Hot Kafka, Alibaba Self-developed notify, Metaq, ROCKETMQ and so on. This article does not cover all of these Message Queuing features, but rather explores the important aspects that you need to think about and design when you design a message queue for self-development. In the process we will refer to the many important ideas of these mature message queues.

This article will first explain when you need a message queue, and then take the push model as the main, from the zero-based analysis of the design of a message queue to consider the problem, such as RPC, high availability, sequential and duplicate messages, reliable delivery, consumer relationship resolution.

The advantages of the pull model represented by Kafka are also analyzed. Finally, some advanced topics, such as the use of bulk/asynchronous to improve performance, pull model of the system design concept, storage subsystem design, flow control design, the implementation of fair scheduling. The last four aspects will be explained in the next article.

When Message Queuing is required

When you need to use Message Queuing, you first need to consider its necessity. There are many scenarios where MQ can be used, and the most common ones are doing business decoupling/FINAL consistency/broadcast/fault peak flow control. Conversely, RPC appears more appropriate if strong consistency is required and the results of business logic are concerned. decoupling

Decoupling is the most essential problem to be solved by Message Queuing. The so-called decoupling, simple point is a transaction, only concerned about the core process. Things that need to depend on other systems but are less important can be notified without waiting for results. In other words, the message-based model is concerned with "notifications" rather than "processing."

For example, in the U.S. tour, we have a product center, product Center upstream docking is the main station, mobile backstage, tourism supply chain and other data sources; downstream docking is the screening system, API system and other display systems. When the upstream data changes, if you do not use the message system, it is bound to call our interface to update the data, it is particularly dependent on the product center interface stability and processing capabilities.

But in fact, as the product center of tourism, perhaps only for the tourism self-built supply chain, Product Center update success is their concern. And for the external systems such as group buying, Product Center update success or failure, is not their responsibility. They just need to make sure we're notified when the information changes.

In our downstream, there may be a range of requirements for updating indexes, refreshing caches, and so on. For the product center, this is not our responsibility. Frankly speaking, if they pull the data regularly, it can also guarantee the update of the data, but the real-time sex is not so strong. But using the interface to update their data, obviously for the product center is too "heavyweight", only need to publish a product ID change notification, the downstream system to deal with, it may be more reasonable.

Another example, for our order system, the final payment of the order may need to send users SMS points, but in fact this is not the core process of our system. If the external system is slow (such as a poor SMS gateway), the main process will take longer and the user certainly doesn't want to pay a click to see the results for several minutes. Then we just need to notify the SMS system "we paid successfully", not necessarily wait for it to finish processing. Final Consistency

Final consistency refers to the state of the two systems being consistent, either succeeding or failing. Of course there is a time limit, in theory the sooner the better, but in fact, in the case of various anomalies, there may be a certain delay to achieve the final consistent state, but the last two systems are the same state.

Some of the industry's message queues for "eventual consistency", such as Notify (Ali), QMQ (where to go), are designed to be highly reliable notifications in trading systems.

With a bank transfer process to understand the eventual consistency, the demand for transfer is very simple, if a system to deduct money success, B system plus money must be successful. Instead, they roll back together, like nothing happened.
However, there are many possible surprises in this process:

A deduction money success, call B Plus Money interface failed.

A deduction money success, call B plus money interface although successful, but get the final result when the network exception causes a timeout.

A buckle money success, B plus money failed, a want to back the money, but a machine down machine.

Obviously, it's not so easy to really make this seemingly simple thing. All cross-VM consistency issues, from a technical point of view, a common solution is:

Strong consistency, distributed transactions, but the landing is too difficult and cost too high, the article will be specifically mentioned.

The final consistency is mainly in the form of "record" and "compensation". Before you do all the things that are uncertain, record them and do things that are not sure, and the result may be: success, failure, or uncertainty, and "uncertainty" (such as timeouts) can be equivalent to failure. Success will be able to clean up the records of things, for failure and uncertainty, you can rely on timed tasks and other ways to do all the failure to re-work until the success.

Back to the example, the system in the case of a deduction of money, to give B "notice" this thing recorded in the library (in order to ensure the highest reliability can be notified B system plus money and deduct money success of the two things to maintain in a local transaction), the notification succeeds delete this record, Notification failure or uncertainty relies on the scheduled task to notify us in a compensatory manner until we update the status to the correct state.

The entire model can still be based on RPC, but can be abstracted into a unified model, based on Message Queuing to make an "enterprise bus".

Specifically, the local transaction maintains a business change and notification message, landed together (the failure is rolled back together), and then RPC arrives at the broker, after the broker succeeds, the RPC returns successfully, and local messages can be deleted. Otherwise, local messages are constantly being re-sent by the scheduled task polling, which ensures that the message is reliably landed broker.

The process of sending a message to consumer is similar, and the message is sent until the consumer sends a consumer confirmation. We do not care about the repetition of the message, through two of messages on the ground plus compensation, downstream is bound to receive the message. Then rely on the state machine version number and other ways to do the weighing, update their own business, the final consistency is achieved.

Final consistency is not an essential feature of Message Queuing, but it is true that Message Queuing can be relied upon for eventual consistency. In addition, all message queues that do not guarantee that 100% do not lose messages are theoretically unable to achieve eventual consistency. Well, it should be said theoretically 100%, to rule out serious faults and bugs in the system.

Designs like the Kafka have the possibility of losing messages at the design level (such as timed brush disks, which can drop messages if power is lost). Even if only 1 per thousand of the news is lost, the business must use other means to ensure that the results are correct. Broadcast

One of the basic functions of Message Queuing is to broadcast. If there is no message queue, every time a new business party is connected, we will have to tune up the new interface. With Message Queuing, we only need to be concerned about whether the message is delivered to the queue, as to who wants to subscribe, is downstream, and undoubtedly greatly reduces the workload of development and deployment.

For example, this article began to mention the product center released product changes in the news, as well as a lot of attractions library to re-update the message, there may be "care" side there are many, but the product center and the attraction library only need to publish the change message can, who cares who the access. fault peaks and flow control

Imagine that the downstream is different in the ability to deal with things. For example, the Web front-end of tens of millions of requests per second, not a magical thing, only need to add a little more machines, and then build some LVS load balancer equipment and nginx and so on. However, the processing capacity of the database is very limited, even if the use of SSD plus sub-table, the processing capacity of single-machine is still at million. Due to the cost, we cannot expect the number of machines in the database to catch up with the front end.

This problem also exists between the system and the system, such as the SMS system may be due to the short board effect, speed card on the gateway (hundreds of requests per second), and the concurrency of the front end is not an order of magnitude. But the user receives the text message in half a minute or so at night, generally does not have the big problem. Without Message Queuing, the complex scenarios of negotiation, sliding windows, etc. between the two systems are not said to be impossible.

However, the exponential growth of system complexity is bound to be stored upstream or downstream, and to deal with a series of problems such as timing and congestion. And whenever there is a gap in processing power, it is necessary to develop a set of logic to maintain this logic. Therefore, the use of intermediate systems to dump the communication content of the two systems, and the downstream system has the ability to process these messages, and then processing these messages, is a relatively common way.

In summary, Message Queuing is not a panacea. RPC is better than Message Queuing for requiring strong transaction assurance and latency-sensitive.

You can use Message Queuing to do things that are trivial or important to others but are not so concerned about yourself.

Message Queuing, which supports eventual consistency, can be used to handle "distributed transaction" scenarios where latency is less sensitive, and may be a better way of handling than bulky distributed transactions.

When there is a gap between upstream and downstream system processing capacity, the use of Message Queuing to make a common "funnel". When the downstream has the capacity to process, then the distribution.

If there are many downstream systems that care about your system's notifications, use Message Queuing decisively.

How to design a message queue

Review

We now have a clear message queue usage scenario, and the next step is to design a message queue for implementation.

The message-based system model does not necessarily require broker (Message Queuing service side). The Akka (Actor model), ZEROMQ, etc. are all based on the message system design paradigm, but there is no broker.

We want to design a message queue, and with broker, there are two things to do:

The dump of a message is delivered at a more appropriate point in time, or a series of means assists the message to end up being delivered to the consumer.

Standardize a paradigm and a common model to meet the needs of decoupling, eventual consistency, and peak error.
Break it up and rub it up. The simplest message queue can be made into a message forwarder that makes RPC two times a time. The sender sends the message to the server (hereinafter referred to as broker), and the server forwards the message to the receiving end, which is simple.

In general, the overall idea of designing a message queue is to build a whole stream of data, such as producer sent to Broker,broker for Consumer,consumer to reply to consumer acknowledgments, broker Delete/backup messages, and so on.

Use RPC to string up data streams. Then consider the high availability of RPC, as far as possible stateless, convenient horizontal expansion.

After considering how to host the message stack and then deliver the message at the right time, the best way to handle the heap is to store, and the storage selection needs to take into account many factors such as performance/reliability and development maintenance costs.

In order to realize the broadcasting function, we must maintain the consumption relations, can use Zk/config server and so on to save the consumption relation.

After completing the above functions, Message Queuing is basically implemented. Then we can consider some advanced features, such as reliable delivery, transactional features, performance optimizations, and so on.

Below we will design the message queue to focus on the module as the main line, interspersed with a number of Message Queuing feature implementation method, to specifically analyze the design to implement a message queue in all aspects. implementing the basic queue functionality RPC Communication Protocol

Just now, the so-called message queue, no external two RPC plus a dump, of course, the consumer end of the consumer confirmation of the situation is three times RPC. Since it is RPC, it will inevitably involve a series of topics, what load balance Ah, service discovery Ah, communication protocol Ah, serialization protocol Ah, and so on. In this piece, my strong suggestion is not to reinvent the wheel.

Leverage the company's existing RPC framework: Thrift, Dubbo, or other custom frameworks. Because of the RPC of Message Queuing, there is no essential difference from normal RPC. Of course, using the memchached or Redis protocol to re-write a set of RPC frameworks is not a must (e.g. Metaq uses its own encapsulated Gecko NIO framework, and Kafka uses a similar protocol). But the realization of cost and difficulty is undoubtedly doubled. Eliminate the extreme requirements for efficiency, and you can use an off-the-shelf RPC framework.

Simply put, the server provides two RPC services, one for receiving messages and one for confirming messages received. And it does not matter which server receives the message and the confirmation message, the result is consistent. Of course, this may also involve cross-IDC service issues. Here and the principle of RPC is consistent, as far as possible priority to choose the room delivery. You may ask, if producer and consumer itself in two room, how to do. First, the broker must ensure that all consumer are perceived to exist. Second, producer as far as possible to choose the room near the good. Highly Available

In fact all the high availability, is dependent on the RPC and storage of the high can be used to do. First look at the high availability of RPC, the United States Regiment based on the Mtthrift RPC framework, Ali's Dubbo, and so on, its own service automatically found, load balancing and other functions. The high availability of Message Queuing, as long as the interface that the broker accepts the message and the acknowledgment message is idempotent, and the consumer of several machines processing the message is idempotent, then the availability of the message queue is forwarded to the RPC framework for processing.

So how to guarantee power and so on. The simplest way is to share storage. Broker multiple machines share a DB or a distributed file/KV system, the processing of messages is naturally idempotent. Even if there is a single point of failure, the other nodes can be immediately overhead. In addition, failover can rely on the compensation of timed tasks, which is a feature that the message queue itself can support naturally. The availability of the storage system itself we don't need to be too paranoid, just give it to the DBAs.

For queues that do not share storage, such as Kafka using Partition plus Master and Standby mode, it is a bit cumbersome. Need to ensure high availability within each partition, that is, each partition must have at least one master and need to do the synchronization of data, about this ha details, you can refer to the next Pull model message system design. the ability of the service side to host message stacking

The message arrives at the server without any processing to the receiver, the broker loses its meaning. In order to meet our fault peak/flow control/final reach, such as a series of requirements, the message stored down, and then choose the timing of delivery is logical.

Just this storage can be made in many ways. For example, stored in memory, stored in the distributed kv, stored in the disk, stored in the database and so on. But it comes down to the persistence and non-persistence of the two main types.

The form of persistence ensures greater reliability of the message (such as a power outage, etc.) and can theoretically carry a greater amount of message buildup (external memory is much larger than memory).

But not every message requires persistent storage. Many messages have a higher demand for delivery performance than reliability, and are large in number (such as logs). At this time, the message does not fall directly into the memory, try a few failover, the final delivery can not be.

Message Queuing is generally supported in two forms on the market. Of course, the specific scenario should be specifically combined with the company's business view. selection of storage subsystems

Let's take a look at the choice of various storage subsystems if data is required to land on the ground. Theoretically, in terms of speed, file systems > Distributed KV (persistent) > Distributed File Systems > databases, while reliability is diametrically opposed. Or to make the most reasonable choice from the supported business scenarios, if your message queue is used to support payment/trading and so on reliability requirements are very high, but the performance and volume requirements are not so high, and there is no time to devote to the study of file storage System, DB is the best choice.

However, DB is subject to IOPS, and file-based storage is a good solution if you require a single broker over 5-digit QPS performance. The whole can be processed by the way of data file + index file, the design of this block is more complex, can refer to the next chapter of the storage subsystem design.

Distributed kv (such as mongodb,hbase), or persistent redis, is a good choice for scenarios where reliability requirements are not so high, due to its friendly programming interface and considerable performance. analysis of consumption relationship

Our message queue now has the ability to dump messages initially. The next important thing is to parse the send and receive relationship and make the correct message delivery.

The message queue on the market defines a bunch of confusing nouns, such as exchange in the Topic/queue,kafka inside the JMS specification, and so on TOPIC/PARTITION/CONSUMERGROUP,RABBITMQ. It is the difference between unicast and broadcast that the nature of the phenomenon can be left out. The so-called unicast, is point-to-points, and broadcast, is a little to multipoint. Of course, for most applications on the Internet, inter-group broadcasts and intra-group unicast are the most common scenarios.

Messages need to be notified to multiple business clusters, and a business cluster has many machines, as long as a single machine consumes the message.

Of course, this is not absolute, many times in the group of broadcast is also a suitable scenario, such as the local cache updates and so on. In addition, the consumption relationship may have multi-level tree relationships in addition to groups within the group. This situation is too complex to be considered in the general context. Therefore, the general comparison of the general design is to support inter-group broadcasting, different groups to register different subscriptions. The different machines within the group, if registered with an identical ID, are unicast, or broadcast if different IDs (such as IP address + port) are registered.

As for the maintenance of broadcast relations, generally because the message queue itself is a cluster, it is maintained on public storage, such as config server, zookeeper, and so on. The things to do to maintain a broadcast relationship are basically the same:

The maintenance of the sending relationship.

Notification when a relationship change is sent.

Queue advanced feature Design

These are some of the basic features of Message Queuing, and here are some things related to Message Queuing features, regardless of the reliability of delivery/message loss and duplication and transactional to performance, not every message queue will be taken care of, so according to business needs, to carefully measure the cost of various features realized, pros and cons, Finally make the most reasonable design. Reliable delivery (final consistency)