The ultimate consistent MQ implementation of distributed transactions

Original link: http://www.jianshu.com/p/716d3ec567c0 The origin of the problem Distributed System characteristics

The readers who have studied the distributed system may have heard of the "Cap law", "Base theory", and so on, it is very coincidence that the acid in chemical theory is sour, base is exactly alkali. We do not have much explanation for these concepts here, and interested readers can look at the relevant references.

Here for consistency we do a simple science:

There are three types of distributed transactions with strong consistency, weak consistency, and eventual consistency:

Strong consistency

When the update operation is complete, access to any number of subsequent processes or threads will return the most recent updated value. This is the user's most friendly, that is, what the user wrote last time, next time will be guaranteed to read what. According to the CAP theory, this implementation requires sacrifice of availability.

Weak consistency

The system does not guarantee that the continuation process or access to the thread will return the most recent updated value. After the data has been written successfully, the system does not commit to immediately read the latest written value, nor does it specify how long it will be read.

Ultimately consistent

The specific form of weak consistency. The system guarantees that, without subsequent updates, the system eventually returns the value of the last update operation. In the absence of failure, the time of the inconsistency window is mainly affected by the delay of communication, the number of system load and replica. DNS is a typical final consistency system. In

distributed systems, it is almost impossible to meet the "consistency", "Availability" and "Partition Fault tolerance" in the "Cap law". In the vast majority of the Internet scene, the need to sacrifice strong consistency in exchange for the high availability of the system, the system often only need to ensure "final consistency", as long as the final time is acceptable to the user in the range, we only need to use short-term data inconsistency can achieve the effect we want.

Example Description

For example, there are orders, inventory of two data, a process to simplify the order, add one orders, minus one inventory. While orders and inventory are independent services, how to ensure data consistency.

At this time we need to think about how to ensure that two remote calls "simultaneous success", the data consistent.

Please note that the most depressing thing about remote invocation is that there are 3 kinds of results, successes, failures, and timeouts. A successful failure is possible if it is timed out.

General solution, most of the practice is to use MQ to make the final agreement. how to achieve eventual consistency

Example Analysis

How do we use MQ to achieve the final agreement? Let's take a detailed analysis together:

Order Business Analysis

First, take the example of the order business we mentioned above:

Insert Loga While we are adding orders (this process can be done on a local transaction)
You can then read Loga asynchronously, sending MQA
B-end receive MQA, while reducing inventory, B here need to do idempotent (avoid business confusion caused by duplicate messages) complex mixed asynchronous business calls

Then we may associate such a problem with the above analysis.

Local execution of the transaction, the execution of the success of a message in the past, the consumer to get the message to do their own business.
For example, a,b,c A calls b,c asynchronously, if B fails, or b succeeds, or b times out, then how do you use MQ to make them last? b failure failed, b after the success of a message to C, B and C for a is asynchronous, and they are all at the same time, and need to a,b,c simultaneous success situation, then this situation with MQ how to do?
In fact, the practice still refers to the concept of local affairs.

The first scenario: assuming that all three of the a,b,c are performing normally, the entire business ends normally.

Second case: Suppose B timed out, then need A to B to re-send the message (remember B service to do power, etc.), if there is a failure, need to see the situation, is Terminal Services, or continue to re-send, even human intervention (all rules need to be based on business rules to set)

A third scenario: suppose one of the three a,b,c fails, the failed service uses MQ to send messages to other services, other services receive messages, query local transaction log logs, and if local fails, delete the received message (indicating that the message consumed successfully), if local success, The compensation interface needs to be called for compensation (a business compensation interface is required for each service). Precautions

MQ here has a pit, usually only to allow the first operation to fail the scene, that is, the first success must be guaranteed after the operation of the business is no barrier, or the subsequent failure of the front bad rollback, only allow the system exception failure, not allow business failure, Usually after a business failure is basically not likely to succeed, if the network or downtime caused by the failure can be resolved by retry, if the business is abnormal, then you can only send messages to a and C let them compensate it. Usually compensated by a third party, ABC provides a compensating interface, and the design paradigm typically does not allow consumption downstream business to fail

How do we understand the above, for example:

For example, a to B transfer, a first to deduct money, and then sent a message, b this side if before this, the number of retries also useless, only manual intervention. Ali's approach to distributed transaction adoption

Ali part of the business is to achieve the ultimate consistency with MQ, but also a part of the business with the TCC transaction, but the TCC transaction with less, because will infect the business, the development cost is higher, if the volume of small words directly with JTA or MQ support transaction is good, in fact, in the distributed transaction this piece has a maximum effort type, A way to be more brain-free.

Wen/jsondream (author of Jane's book)