Introduction to Getting Started with ROCKETMQ transaction messages

Description

Friday, the time of the article: Rocketmq4.3 support business!!! , taking advantage of the relevant content at the weekend, the following main content is about ROCKETMQ affairs related content introduced.

Description: Today This article is only introductory, and does not involve a lot of details, first the approximate process to understand, followed by detailed details of the opening instructions.

Theme

• leads to distributed transaction-related content.
• ROCKETMQ the transaction message.
• ROCKETMQ How transactional messages are used.
• How the ROCKETMQ transaction message is implemented.
• Why a transaction message is required to check the mechanism.
• ROCKETMQ is going to do a business message to check.
• What are the limitations of ROCKETMQ for distributed transaction Resolution? As well as instructions.

Elicit distributed transaction-related content

The main thing here is to illustrate under what background this kind of problem arises.

First of all, let's talk about business , say business, the first thing I think of is when I was in college, the teacher often examples of the transfer of things, such as:

Bank Transfer! Zhang San 100 to John Doe's account, which actually requires two SQL statements:

• Minus 100 yuan for Zhang San's account.
• Add 100 yuan to John Doe's account

If the execution of the first SQL statement succeeds and the program is interrupted before executing the second SQL statement (possibly throwing an exception or other reason), then John Doe's account does not add $100, while Zhang San subtracts $100. This is definitely not going to work!

You probably already know what a business is now! Multiple operations in a transaction are either completely successful or fail completely! There can't be half a success! That is, to Zhang San account minus 100 yuan if successful, then to John Doe account plus 100 Yuan operation must also be successful, otherwise give Zhang San minus 100 yuan, and to John Doe plus 100 Yuan is a failure!

If we use spring in a single service, we use an annotation to solve the problem (@Transactional annotations are solved, basically there is nothing going on. ）

With the increasing complexity of services and the increasing volume of data, companies are slowly introducing technologies such as microservices and sub-database sub-tables, which are really useful, but have a problem with dealing with issues.

Emphasis: Distributed transactions, we are generally stressed the ultimate consistency, rather than strong consistency!!!

The main generalization is 3 categories:

• Based on a single JVM, database sub-Libraries are divided into tables (across multiple databases).
• Based on multiple JVMs, the service is split (not across databases).
• Based on multiple JVMs, the service is split and the database is divided into tables.

It is in order to solve the above 3 types of problems that the distributed transaction-related technologies are introduced to solve these problems.

There are a lot of distributed transaction solutions, and the ROCKETMQ transaction messages that need to be addressed in this article are just one of those solutions.

ROCKETMQ Transaction Messages

Here is the main reference ROCKETMQ official documents inside the content, reference address:

http://rocketmq.apache.org/rocketmq/the-design-of-transactional-message/

Half (Prepare) Message

Refers to a message that is not delivered, the sender has successfully sent the message to the MQ server, but the server did not receive two acknowledgements from the producer, at which point the message is marked as "Pending" status, in which the message is a half message.

Message back

Due to network Flash, producer application Restart and other reasons, resulting in the loss of two acknowledgment of a transaction message, the MQ server through the scan to find a message long-term "half-message", the need to proactively ask the message producer the final state of the message (Commit or Rollback), the process is a message back-check.

Execution flowchart

1. Sends a message to the MQ service side to send the direction.
2. After MQ Server persists the message successfully, the message is half-message to the sender ACK acknowledgement that the message has been sent successfully.
3. The sender begins to execute the local transaction logic.
4. The sender submits two acknowledgements (commit or Rollback) to the MQ server based on the results of local transaction execution, and the MQ server receives a commit state to mark the half message as undeliverable, and the Subscriber will eventually receive the message; MQ server receives the Rollback status The half-message is deleted and the Subscriber will not accept the message.
5. In the special case of a network disconnection or application restart, the two acknowledgements that were submitted in step 4 above are eventually not reached by MQ server, and after a fixed time, MQ server initiates a message back to the message.
6. After the sender receives the message back, it needs to check the final result of the local transaction execution of the corresponding message.
7. The sender again submits two acknowledgements based on the final state of the local transaction that was checked, and MQ Server still operates half-message in step 4.

The transaction message is sent for steps 1, 2, 3, 4, and the transaction message is traced back to steps 5, 6, 7.

ROCKETMQ how transaction messages are used 1, introduce rocketmq-client

There is no, wait a few days will have, or download the source of their own packaging to their own private warehouse is the same, after the introduction of 4.2.0 and almost, is replaced by 4.3.0.

<dependency>    <groupId>org.apache.rocketmq</groupId>    <artifactId>rocketmq-client</artifactId>    <version>4.3.0</version></dependency>

2, write producer

The main reference is RocketMQ4.3.0 inside the example, the address is:

Github.com/apache/rocketmq/tree/release-4.3.0/example/src/main/java/org/apache/rocketmq/example/transaction

Note: Unlike normal message sending Defaultmqproducer , this is the use of transactionmqproducer.

key points: implement the Transactionlistener interface yourself, and implement the Executelocaltransaction method (which performs local transactions, typically the operation of DB-related content) and The Checklocaltransaction method , which is used to provide the broker with the ability to check local transaction messages, stores the results of the local transaction execution in Redis or DB, and prepares the data for it.

Or to: to Zhang San account minus 100 yuan, to John Doe account plus 100 yuan for example. This time we send the equivalent to do is Zhang San account minus 100 yuan operation, and the operation of the DB successfully, the following is another John Doe account plus 100 Yuan operation , this time is equivalent to the definition just sent the transaction message topic, There is no difference between the consumer and our normal consumer side.

Think: What if we fail the consumer at this time? (This issue followed by discussion, there are 2 of consumption failures, the first is time-out, we can retry, the second is the real processing failure?) What should we do? ）

How the ROCKETMQ transaction message is implemented

The first sensation is very similar to the timed message approach, but more complex than timed messages. Timed message content in: ROCKETMQ (ix): Message sent (continued) mentioned in.

essence: The timing message first changes the TOPIC of the timing message to SCHEDULE_TOPIC_XXXX, followed by a series of processing ..., we also have this transaction message, first send the half (Prepare) message Message, in fact TOPIC content also continued to modify (rmq_sys_trans_half_topic), all consumer is not visible (if the submission is true TOPIC, If you need to roll back then the temporary topic content is deleted. ）

half (Prepare) Message modifies the topic breakpoint as follows:

This is just an introductory introduction, which will be analyzed in the subsequent chapters of the detailed details.

think: if the sending of ordinary messages and local execution logic in a transaction, if the execution of a transaction succeeds in sending ordinary messages, if the failure to roll back as if it is possible, then this transaction message relative to its advantages or benefits??? Think about it.

Why a transaction message is required to check the mechanism

In fact, this content in,ROCKETMQ transaction information inside also explained that, because of the network Flash, the producer application restarts and other reasons, resulting in a transaction message two acknowledgment is lost, the MQ service side through the scan to find a message long-term "half message", You need to proactively ask the message producer about the final state of the message (Commit or Rollback), which is a message back-check. So what exactly does ROCKETMQ do? The following will be introduced shortly.

ROCKETMQ is going to do a business message.

The TOPIC subject of the half (Prepare) message is checked for rmq_sys_trans_half_topic messages per 60s.

The broker invokes its own Checklocaltransaction method that implements the Transactionlistener interface.

Note: It is ROCKETMQ has implemented this mechanism, today this is only introductory introduction, and does not involve a lot of details, first of all the process is clear, follow-up details of the opening instructions.

What are the limitations of ROCKETMQ for distributed transaction Resolution? and instructions

What do we say if the consumption fails when the ROCKETMQ transaction message is used ? There are 2 consumption failures, the first is time-out, we can try again, the second is the real processing failure? Think carefully, this piece is still quite complex, if need to have 7-8 business module, which executes to 6th business module to fail? this retry several times or failed, how should we deal with it??? do you want to roll back the previous 5 actions? So complex and complex, why ROCKETMQ not provide automatic rollback? I hope you think, if the failure rate is large, then the system problem needs to optimize the code business ...

According to the degree of zero understanding, a lot of this is through manual intervention and t+1 Reconciliation and compensation mechanism.

Conclusion

My level is limited, inevitably there will be some understanding of the deviation of the place, if found, welcome to actively point out, thank you!!!

If you feel that there is a harvest, welcome to praise, attention, add the public number "ingenuity Zero", read more wonderful history!!!