The realization principle of the distributed database middleware DDM

As data volumes continue to grow, traditional architectural patterns are difficult to solve, especially when business is linear or even exponentially higher. At this point we have to expand horizontally, the database to the different servers to solve the problem, which is what we call the database middleware.


As a database middleware, the distributed database middleware DDM manages the underlying database storage engine in a cluster way, which is very convenient for users. The application does not need to be concerned about how many shards are specific. Similar to the operation of a single database, the user through the DDM Management console for database operations, the use of JDBC and other driver services or SQL Client connection database, data read and write.

Diagram of the business architecture of the DDM service

Sharding is a direct way to address the limits of database storage capacity. Shards include both vertical and horizontal shards.

Vertical shards

Vertical fragmentation is also called vertical segmentation, that is, in logical tables, the original database is cut into multiple databases. After slicing, different tables are stored on different databases.

Vertical sharding is closely linked to business architecture design. For example, from the business domain of the system architecture optimization, divided into a number of sub-business systems, sub-business system coupling is low. Communication and data exchange between sub-business systems by means of interface. The business is clear after vertical splitting, the splitting rules are clear, and the system is easy to integrate and expand. Typically used for database upper-level architecture design.

Vertical shards

Horizontal sharding

Horizontal sharding is called horizontal segmentation, that is, the data rows in the logical table are recorded as units, the original logical database is divided into multiple physical database shards, the table data record distribution is stored on each shard.

Horizontal sharding mainly uses the business architecture cannot continue to subdivide, but in the database the single table data quantity is too big, the query performance degrades the scene. Through horizontal sharding, it solves the problem of single-library capacity and improves concurrent query performance.

Horizontal sharding

DDM enables automatic horizontal sharding, where the application does not need to be concerned about which piece of data is stored on the Shard. The level of the logical table is required according to a certain shard rules, such as an order tracking system (see), we select the order number (ORDERID) as a split key, respectively, "Order Flow table", "Order Details table" and "Logistics tracking table" horizontal split, split rules for the key value hash after the mold, The Shard calculation rule is as follows:

H (Key (OrderId)) = Hash (Key (OrderId))%N

where n denotes a total of n data shards, H (Key (OrderId)) represents the Shard number that the order is stored after the order number is hashed and modulo is evaluated.

Data storage after fragmentation

Routing distribution
Routing distribution is the same as the basic functionality of DDM with horizontal sharding. In the distributed database, the function of the route is to parse the SQL statement and forward it to the correct shard, ensure the correct result after SQL execution, and save the QPS resources. For example: The Order payment system contains shard0, Shard1, Shard2 three shards, order number 2017010112345678 of the order data stored on the Shard0 shard, the following statement should be routed to the shard0 shard execution.

Select Customer, Orderstatus, createdate from Order

where OrderId = ' 2017010112345678 ';

If you route to Shard0, Shard1, Shard2 three shards at the same time, it can cause redundant queries, waste resources, and if routed to Shard1, Shard2 shards, you will not get the correct return result.

The routing parsing process for a single sheet of DDM is as follows:

Route resolution process for single-sheet tables

Read/write separation

A database that consumes more compute and cache resources is often a dense or complex SQL query. When the system resource is consumed by the query statement, which in turn affects the data write operation, the overall performance of the database is degraded and the response is slow. As a result, you can add a read-only instance to a database when the database CPU and memory resources are high and the read-write ratio is higher.

Adding a read-only instance has the following effects:

1, the query non-transactional query SQL routing to a read-only instance execution, the primary instance of the execution of transactional SQL, to a large extent to alleviate the primary instance of the S-Lock and X-Lock competition.

2, the table on the read-only instance can be configured to not provide transaction support database engine, and thus improve query efficiency.

3, increase the read-only instance, also equivalent to the database scale-out, directly increase the load capacity, while increasing data redundancy, to ensure that the database is highly available.

The DDM service enables automatic read and write separation, when the user buys an RDS read-only instance, synchronizes the read-only instance information to the DDM, eliminating the need for additional configuration. At the same time, DDM supports the user to customize the read and write separation policy in SQL, see how to implement read/write separation.

Read/write separation

Smooth expansion

With the growth of the business, the logical library storage space is not enough, the concurrency pressure is large, at this time the DDM instance logical library can be scaled up, by increasing the RDS instance to improve the data storage capacity and concurrency support capability.

Smooth expansion is a way of horizontal expansion, by increasing the number of RDS instances to increase the overall data storage capacity, the sub-database is smoothly expanded to the new RDS instance, to ensure that all the data are distributed evenly on each sub-library, reducing the processing pressure of a single RDS instance. The principle of smooth scaling is as shown.

Principle of smooth expansion

Above is the analysis of the principle of the distributed database middleware DDM, Huawei Cloud DDM has launched the free experience activities, want to learn more, welcome to the distributed Database Middleware view.

The realization principle of the distributed database middleware DDM