Distributed Message Queuing Rocketmq&kafka--"sequential consumption" of messages--a seemingly simple complex problem

When it comes to messaging middleware, we usually talk about a feature: the sequential consumption of messages. The problem seems simple: Producer sends messages 1, 2, 3 ... Consumer Press 1, 2, 3 ... Sequential consumption.

But the reality is: no matter rocketmq, or Kafka, the default does not guarantee the strict and orderly consumption of messages.

This feature looks simple, but why is it not guaranteed by default?

Interested friends can pay attention to the public number "the way of architecture and technique", get the latest articles and further discussion.
or scan the following QR code:
how difficult it is to "strictly order consumption"

The following is an analysis from 3 aspects, for a message middleware, "strict sequential consumption" how difficult, or impossible. Send Side

The sending side cannot send asynchronously, and the asynchronous send cannot guarantee the message order in the case of failed send.

For example, you made a series of three. After a while, the results returned 1 failed, 2, 3 succeeded. You send the 1 again 1 times, this time the order will be messed up. Storage-side

For the storage side, there are several issues to ensure the message order:
(1) The message cannot be partitioned. That is, 1 topic, only 1 queues. In Kafka, it is called partition; in Rocketmq, it is called a queue. If you have multiple queues, the same 1 topic messages will be scattered across multiple partitions, naturally not in the order of guarantee.

(2) Even if there are only 1 queues, there will be a 2nd problem. After the machine has been hung, can I switch to another machine? This is the high availability issue.

For example, your current machine is hung up, and there is still news on it that is not consumed. When switching to another machine, availability is guaranteed. But the message order was messed up.

To ensure that on the one hand synchronous replication, not asynchronous replication, and the other 1 to ensure that the machine before cutting, the hanging off the machine above, all the messages must be consumed, there is no residue. Obviously, this is hard ... Receiving End

For the receiving end, can not be consumed in parallel, that is, can not open multi-threaded or multiple clients consume the same 1 queues. Summary

From the above analysis can be seen, to ensure that the message is strict and orderly, how difficult.

The issue of the sending and receiving side, fortunately to solve a point, limit the asynchronous send, limit the parallel consumption. But for the storage side, after the machine hangs, the problem of switching, it is difficult to solve.

You switch, the message may be messy, and if you don't switch, it's temporarily unavailable. There is a tradeoff between those 2. does the business need to be global in order.

As can be seen from the above analysis, to ensure a topic inside, the message is strictly ordered, is very difficult, or the condition is very harsh.

What to do then. We must use all our strength and every means to ensure that the message is strictly ordered.

This is a question that needs to be considered from a different perspective: the business perspective. As described in the following blog post:
http://www.jianshu.com/p/453c6e7ff81c

In fact:
(1) There is a large number of businesses that do not care about order;
(2) The unordered queue does not mean that the message is unordered.

The meaning of paragraph (2) is that we do not guarantee the overall order of the queue, but we can guarantee the local order of the message.

For example: Ensure that messages from the same 1 order IDs are ordered.

Here's how to deal with this problem in Kafka and ROCKETMQ, respectively:

Kafka: When sending 1 messages, you can specify 3 parameters (topic, partition, key). Partiton and key are optional.

If you specify partition, all messages are sent to the same 1 partition, which is ordered. And on the consumer side, Kafka guarantee that 1 partition can only be consumed by 1 consumer.

Or, if you specify a key (such as an order ID), all messages with the same 1 key will be sent to the same 1 partition. is also orderly.

ROCKETMQ:ROCKETMQ, on the basis of Kafka, has eased this restriction a step further. Specifies only (topic, key), and does not specify which queue to go to specifically. In other words, it does not want the business side, not to want a global strict and orderly.

Key point: This release, in fact, involves a bigger problem. Is the significant difference between ROCKETMQ and Kafka on the underlying storage. I have introduced this in 1, the sequel to "Righting The wrongs".

Later in the source analysis sequence, this problem is further analyzed.

On the issue of "message order", this is where the discussion ends.