Database Sub-Library sub-table transaction solution

I. Overview

With the development of time and business, the data volume of the table in the database will be more and more large, correspondingly, the data operation, adding and deleting the cost will be more and more big. Therefore, some of these large tables are split into multiple tables in several databases.
On the other hand, it is necessary to guarantee the transactional consistency of the sub-database and the main repository after the sub-database table.

Ii. need to solve the problem 2.1 transaction issues

Since the database is divided into tables, how to ensure the transaction of the main library and the repository is the problem that must be solved in the new table. By creating a water table in the main library, the logic of the operational database is mapped to a running record. When the entire large transaction is completed (the stream is inserted into the water table), then the flow is executed in other ways to ensure eventual consistency.

2.2 Water

The above by creating a water table in the database, using a running record represents a business processing logic, therefore, a water must be able to finally execute correctly. Therefore, when a piece of business code extraction flow must take into account:

1. Flow delay processing. Water is not processed in real time, but is executed asynchronously using a running water actuator. Therefore, if you are in the original logic, you need to pay particular attention to whether the downstream process has real-time dependencies on the pipelining (for example, the subsequent business logic uses pipelining results to do some calculations, etc.).
2. The flow of water to deal with disorder. Ensure that even after the generated streams are executed first, there is no problem.
3. The ultimate success of running water. For each inserted stream of water, the water must be guaranteed to perform successfully

Therefore, when extracting the water:

1. The simpler the flow, the better.
2. Less dependence on lost processing, better
3. The extracted pipelining has no real-time dependency in the business logic.

2.3 Water Processor

The flow processor is to ensure that the flow processing as fast as possible, but also to ensure that the flow of water can ultimately be successful.

Imagine a scenario: when the occurrence of a certain stream processing failure, if the drain executor to wait for the current execution of running water to continue to execute, it will affect the execution of subsequent pipelining, more serious is stuck in the record, causing the whole system problems

As a result, 2 tasks are set in the running water actuator:

1. The first task, water Processing task, has the fastest speed of execution of water, if the water treatment failed, also does not affect the back flow water treatment
2. The second task, the Water Check task, this task is to check the running records sequentially, to ensure that all the running water is successful, if the failure, retry, multiple retries failed to issue an alarm to let the manual intervention processing.

2.4 How to indicate that the pipelining process is complete

Because the water table is placed in the original database, and after the flow of water treatment is the operation of the sub-Library, how to ensure that the flow of state updates and sub-Library is also in a transaction?

Create a water table in the sub-Library, when the drain processing is completed, not to update the cousin State, but to insert the sub-database flow table,

So as long as the original database and the sub-Library of water, you can process the results of water

If all the running water is successful, then the Flow table records in the main library, and there are corresponding

Three, the basic framework of water processor

The pipelining processor does not actually contain any business-related processing logic, and the core functions are:

1. Notifies the business provider when to deal with what kind of water
2. Verify the success of running water execution

Note: The running water executor does not know what logic the flow represents, and it needs the business system to recognize and execute the relative business logic.

3.1 Running Water execution task

Water processing scheduling task is to scan the pipeline to be processed, and then notify the business system to execute which stream.

As follows:

3.2 Water Check Task

The running water Check task is to compare the records of the streams in the main library and the sub-Library, and to re-process the unsuccessful flow notification business system and issue an alarm if multiple retries fail.

Process:

Database Sub-Library sub-table transaction solution