How to replace a distributed transaction with Message Queuing

Transferred from: http://blog.csdn.net/ss123465/article/details/7964184

Because of the huge amount of data, most Web applications need to deploy many database instances. In this way, some user actions may need to modify data in multiple DB instances. The traditional solution is to use distributed transactions to ensure global consistency of data, and the classic approach is to use the two-phase commit protocol.

For a long time, the elegant global acid that is provided by distributed transactions guarantees that the application developer's mind is anesthetized, and many people are downside to imagine what a world without distributed transactions would be like. Today, like MySQL and PostgreSQL, the open source database for low-end users supports distributed transactions, and developers revel in it, regardless of whether distributed transactions harm the system.

In fact, there is a loss, and the acid guarantee provided by distributed transactions is at the cost of compromising the availability, performance, and scalability of the system. Distributed transactions can only be completed successfully if each database instance participating in the distributed transaction is functioning properly, and the entire transaction cannot be completed as long as one is not working properly. In this way, the availability of the system is equal to the availability of each instance participating in the distributed transaction, the more instances, the more obvious the decrease in availability. From the perspective of performance and scalability, the first is that the total duration of a transaction is usually the sum of the operation time of each instance, because each operation in a transaction is usually executed sequentially, so that the response time of the transaction is increased, followed by the general web application of the transaction is not small, the single machine operation time of a few milliseconds or less than 1 milliseconds When it comes to distributed transactions, the network traffic round-tripping between points in the submission time is also the millisecond level, and the impact on transaction response times cannot be ignored. As transaction duration is prolonged, transaction lock-in time for related resources increases correspondingly, which can significantly increase concurrency conflicts and affect system throughput and scalability.

Because of the problem of distributed transaction, ebay does not adopt distributed transaction in design, but solves data consistency problem by other means. The most important techniques used are Message Queuing and message application state tables.

Let me give you an example. Suppose the following two tables are in the system
User (ID, name, amt_sold, Amt_bought)
Transaction (XID, seller_id, buyer_id, amount)
Where the user table records users ' transaction summary information, the transaction table records the details of each transaction.

Thus, in the case of a transaction, if you use a transaction, you need to do the following with the database:
Begin
INSERT into Transaction VALUES (XID, $seller _id, $buyer _id, $amount);
UPDATE user SET Amt_sold = amt_sold + $amount WHERE id = $seller _id;
UPDATE user SET amt_bought = amt_bought + $amount WHERE id = $buyer _id;
Commit
The transaction information is recorded in the transaction table, and then the status of the seller and the buyer is updated.

Assuming that the transaction table and the user table are stored on different nodes, the above transaction is a distributed transaction. To eliminate this distributed transaction, split it into two sub-transactions, an update to the transaction table, an update to the user table is not possible, because there is a possibility that the transaction table update succeeds, the update user fails, the system will not be able to revert to a consistent state.

The solution is to use Message Queuing. As shown below, starting a transaction, updating the transaction table, does not directly update the user table, but instead inserts an update to the user table into the message queue. There is also an asynchronous task to poll the contents of the queue for processing.
Begin
INSERT into Transaction VALUES (XID, $seller _id, $buyer _id, $amount);
Put_to_queue "Update user (" seller ", $seller _id, amount);
Put_to_queue "Update User (" buyer ", $buyer _id, amount);
Commit
For each message in queue
Begin
Dequeue message;
If Message.type = "Seller" Then
UPDATE user SET Amt_sold = amt_sold + message.amount WHERE id = message.user_id;
Else
UPDATE user SET amt_bought = amt_bought + message.amount WHERE id = message.user_id;
End
Commit
End

The solution looks perfect, and it doesn't actually solve the distributed problem. In order for the first transaction to not involve a distributed operation, Message Queuing must use the same set of storage resources as the transaction table, but in order for the second transaction to be local, the message queue store must be associated with the user table. Neither of these can be satisfied at the same time.

If the message has the power of operation, that is, a message is applied more than once and the effect of the application is the same, the above problem is well resolved, as long as the message queue is placed in the transaction table together, and then in the second transaction, the message is first applied, and then removed from the message queue. Because Message Queuing storage is not associated with the user table, the system fails after the message is applied and it may not have time to remove the applied message from the queue. This message will be re-applied once the system resumes, and the application can produce the correct results many times due to the idempotent nature.

But in practice, the message is very difficult to be idempotent, such as the above update operation, the execution and execution of the end of multiple times is obviously not the same. The solution to this problem is to use another table to record messages that have been successfully applied, and the table uses the same storage as the user table. Assuming that the following table Message_applied (msg_id) records the messages that were successfully applied, the resulting solution is as follows:
Begin;
INSERT into Transaction VALUES (XID, $seller _id, $buyer _id, $amount);
Put_to_queue "Update user (" seller ", $seller _id, amount);
Put_to_queue "Update user (" buyer ", $buyer _id, amount);
Commit;
for each message in the queue
begin;
SELECT Count (*) as CNT from message_applied WHERE msg_id = message.id;
If cnt = 0 Then
if message.type = "Seller" then
UPDATE user SET amt_sold = amt_sold + message.amount WHERE id = message.user_id;
Else
UPDATE user SET amt_bought = amt_bought + message.amount WHERE id = message.user_id;
End
INSERT into message_applied VALUES (message.id);
End
Commit;
If the above transaction succeeds
Dequeue message
DELETE from message_applied WHERE msg_id = message.id;
End
End

Let's take a closer look:
1, Message Queuing and transaction use the same instance, so the first transaction does not involve distributed operations;
2, message_applied and user table in the same instance, can also ensure consistency;
3, after the end of the second transaction, Dequeue message before the system may fail, after the failure of the system will re-remove the message from the message queue, but through the message_applied table can be checked out the message has been applied, skipping this message to achieve the correct behavior;
4, finally will have been successfully applied, and has been removed from the message queue deleted from the Message_applied table, you can ensure that the message_applied table in a very small state (not clear is also possible, does not affect the system correctness). Because Message Queuing and message_applied are on different instances, dequeue message will fail before the corresponding message_applied record is deleted. Once the failure occurs, the Message_applied table will leave some garbage content, but does not affect the correctness of the system, in addition to the garbage content can be properly cleaned.

Although there is no strong consistency guarantee for distributed transactions, the system will be in an inconsistent state for a short time in case of system failure with the above scheme. However, based on Message Queuing and message application status tables, the system can eventually be restored to a consistent state. With the Message Queuing scheme, the tight coupling between the two DB instances is lifted, and the performance and scalability of the database are not comparable to the distributed transaction.

Of course, using distributed transactions helps simplify application development, and using Message Queuing significantly requires more work, both of which have advantages and disadvantages. The personal point of view is that for a system with a tight time or low performance requirement, distributed transactions should be used to speed up development efficiency, and for systems with high performance requirements that are not very tight in time, you should consider using Message Queuing scenarios. For systems that use distributed transactions and have a stable system with high performance requirements, you can use Message Queuing schemes to refactor to optimize performance.

Note: This article is based on the article written by the ebay engineer Dan Pritchet

(go) How to replace distributed transactions with Message Queuing