Implementing distributed transactions using event and Message Queuing

Original: http://skaka.me/blog/2016/04/21/springcloud1/

Unlike a single-schema application (Monolith), transactional operations can become difficult in a distributed environment because distributed environments often have multiple data sources, and it is difficult to ensure consistency of multiple data source data with local database transactions. In this case, a two-phase or three-phase commit protocol can be used to complete a distributed transaction. However, in general, the performance is poor in this way because the transaction manager needs to wait several times between multiple data sources. There is a way to solve the distributed transaction problem, and the performance is good, this is my article to describe the use of events, local transactions and Message Queuing to implement distributed transactions.

Let's start with a simple example. Basic all Internet applications will have the user registration function. In this example, we have two steps for user registration:
1. Register successfully, save user information.
2. The user is required to issue a voucher to encourage the user to consume.
For a single architecture application, this is simple: in a local transaction, a record is inserted into the user table, and a record is inserted in the voucher table, and the commit transaction is completed. But if our application is implemented with MicroServices, maybe users and vouchers are two separate services, They have their own applications and databases, so there is no way to simply use local transactions to ensure the atomicity of operations. Now let's look at the use of event mechanisms and Message Queuing to implement this requirement. (The message queue I'm using here is Kafka, and the same principle applies to other queues such as ACTIVEMQ/RABBITMQ)

We will create an event for the user to register this action, which is called the user creation event (user_created). After the user service has successfully saved the user record, a user creation event is sent to the message queue, the voucher service listens for user-created events, and once the event is received, the voucher service creates a voucher for the user in its own database. Well, these steps look pretty straightforward, But how do you guarantee the atomicity of the transaction? Consider these two scenarios:
1. The user Service is down before the user record is saved, before it can be sent to the message queue. How do you ensure that user-created events are sent to Message Queuing?
2. The voucher service receives user-created events and goes down before the event can be processed. How do I consume user-created events before restarting?
The essence of these two issues is how to make the operations database and operation Message Queuing an atomic operation. Regardless of 2PC, here we can solve this problem through the event table. The following is a class diagram.

Eventpublish is a table that records events to be published. which
ID: Each event generates a globally unique ID, such as a UUID, when it is created.
Status: Event state, enumeration type. There are now only two states: Pending release (NEW), published (PUBLISHED).
Payload: Event content. Here we'll turn the event content into JSON and save it in this field.
EventType: Event type, enumeration type. Each event will have a type, such as the one we mentioned earlier to create the user user_created is an event type.
The eventprocess is used to record pending events. Fields are basically the same as eventpublish.

Let's look at the release process of the event first. The following is a sequence diagram of User Service publishing user-created events.
1. The user Service opens a transaction after receiving a user request, creates a user record in the user table, and creates a record of status new in the Eventpublish table, payload records the event content and commits the transaction.
2. The timer in the user Service starts the transaction first, then queries whether the eventpublish has a status of new record, after querying to the record, gets the payload information, publishes the message to the corresponding topic in the Kafka.
After the send succeeds, modify the status of Eventpublish in the database to published and commit the transaction.

The following is a sequence diagram of the voucher service handling user-created events.
1. The voucher service receives the user creation event from Kafka (which is actually the message that the voucher service actively pulls, ignoring the implementation of the message queue first), creating a record of status new in the Eventprocess table, payload the event content, and if the save is successful, Returns a successfully received message to Kafka.
2. The timer in the voucher service starts the transaction first, then queries whether the eventprocess has a status of new record, after querying to the record, gets the payload information, gives the event callback processor processing, here is directly creates the voucher record. After processing succeeds, modify the status of Eventprocess in the database to processed, and then commit the transaction.

Looking back at the two questions we raised earlier:
1. The user Service is down before the user record is saved, before it can be sent to the message queue. How do you ensure that user-created events are sent to Message Queuing?
Based on the sequence diagram of the event release, we split the creation and release events into two steps. If the event was created successfully, but it was down when it was released. After startup, the timer will re-publish the event that was not previously published successfully. If the event was created, it was down because event creation and business operations were in a database transaction, so the corresponding business operations failed, and the consistency of the database state was ensured.
2. The voucher service receives user-created events and goes down before the event can be processed. How do I consume user-created events before restarting?
Based on the sequence diagram of the event processing, we divide the receiving event and the processing event into two steps. If the event is received successfully, but is down during processing. Once started, the timer will re-process the event that was not successfully processed. If the event is down when it is received, Kafka will resend the event to the corresponding service.

In this way, we do not use 2PC and guarantee the final consistency of state between multiple data sources.
This approach to asynchronous transactions has the advantages that asynchronous systems typically have, compared to the way 2pc/3pc is synchronous transaction processing:
1. Transaction throughput is large. Because there is no need to wait for other data source responses.
2. Good fault tolerance. A service may not even be online at the time the event is published.
Disadvantages:
1. Programming and debugging is more complicated.
2. More intermediate states are prone to occur. For example, the user service has saved the user and released the event, but before the voucher service can be processed, users will find themselves without vouchers if they log into the system. This situation may be tolerated in some businesses, but some businesses are not. So consider it before you develop it.

In addition, the above process in the implementation of the process there are some areas that can be improved:
1. Timers when updating the Eventpublish status to published, you can update the status of multiple eventprocess in one batch.
2. When the timer query is eventprocess and handed over to the event callback handler, the thread pool can be used asynchronously to speed up the eventprocess processing cycle.
3. While saving eventpublish and eventprocess to Redis, subsequent operations can be performed on the data in Redis, but care should be taken to handle cache and database possible state inconsistencies.
4. For Kafka, because the Kafka is characterized by the possibility of re-sending messages, so when the event is received and saved to eventprocess may report the primary key conflict error (because the duplicate message ID is the same), this time can be directly discarded the message.

About 2PC:HTTPS://ZH.WIKIPEDIA.ORG/WIKI/%E4%BA%8C%E9%98%B6%E6%AE%B5%E6%8F%90%E4%BA%A4

Implementing distributed transactions using event and Message Queuing