Dialog DDM: Full analysis of distributed database middleware

Into the cloud computing era, the traditional database in the performance and capacity of the enterprise has been unable to meet the requirements, with the increasing volume of data, easy to expand, split database solutions for the enterprise's cloud transformation is particularly important. In order to make enterprise application on cloud simpler, distributed database Middleware (distributed middleware) focused on solving the database bottleneck problem faced by enterprise in the cloud process, not only can meet the business requirements of horizontal splitting, expansion, read and write separation, etc. It is also more cost-effective than traditional solutions. Next, let's decrypt DDM with 0 distance.

What is DDM?

DDM focuses on solving the problem of database distributed extension, which breaks through the capacity and performance bottleneck of traditional database and achieves high concurrent access to massive data. DDM provides distributed database capabilities for application-transparent database read-write separation, automatic data fragmentation, and flexible elastic scaling.

How does DDM define read-write separations?

From the database point of view, for most applications, from centralized to distribution, the most basic requirement is not the bottleneck of data storage, but the bottleneck of the calculation, that is, the bottleneck of SQL query, in the absence of read-write separation system, it is likely that some complex SQL queries during peak hours will lead to the database system paralysis, From the standpoint of protecting the database, we should try to avoid a single node database without the master-slave replication mechanism. Traditional read-Write separation solution coupling application code, expanding read node or modifying read/write separation policy need to modify the application code, upgrade the application, very complex. DDM realizes the transparent read/write separation, the application realizes the read and write separation does not need to modify the code, in order to guarantee the read consistency, by default in the transaction reads the entire distribution to the master node. Reads the distribution from the node outside the transaction. Writes the distribution Master node. When application requirements are complex, DDM provides read-write separation logic that hint can control SQL by program autonomy. In addition, the back-end DB if part of the node fails, DDM will automatically remove the fault node, automatic master-slave switching, no perception of the application.

(Before and after the transformation of the frame comparison chart)

Application in the micro-service architecture, the service will be split more than the original, and the number of connections to the database will also increase a lot, whether this is also a distributed database middleware need to solve an important problem?

That's right. To give a chestnut, for example, the maximum number of connections for an application is 2000, before service split, the application Exclusive 2000 data connections, assuming split into 100 micro services, then in order to ensure that the total number of connections does not exceed the maximum number of MySQL connections, Then the maximum number of connections that each microservices can configure is 20. This is almost unacceptable for applications. On the market many sub-database sub-table middleware such as Cobar, Atlas, and so on, the backend MySQL connection pool management is based on the Shard to achieve, and does not have the entire MySQL instance sharing interoperability, anti-concurrency capability is severely weakened. While DDM is really based on the MySQL instance pattern, all databases under a MySQL instance share a connection pool. This for the Shard, can avoid some library connection is idle, some libraries are not enough to use the situation, to maximize parallelism. The properties involved in the session level are automatically maintained by DDM and the application is not aware.

Is there a limit on the number of connections in this sharing mode?

The DDM front-end connection is relatively lightweight compared to MySQL connections and can support tens of thousands of connections relatively easily. Of course, to prevent individual users from abusing resources, the maximum number of front-end connection limits is set.

(Before and after the transformation of the frame comparison chart)

In the application scenario, do you have to use the DDM method to solve? There is also a hardware upgrade, the database itself partition scheme, how to choose?

Hardware solutions because of the high cost and poor scalability is not discussed here, and the database itself partition table scheme, can only be confined to a library, the data can not cross the library across the instance, the expansion scheme is limited, DB failure and adjustment need to apply synchronization adjustment, operation and maintenance of difficulty, upgrade and maintain a large workload, small system fortunately, For the large-scale system is unacceptable, it is the solution to adopt distributed database middleware in the long run.

How does DDM do sharding design?

For distributed database middleware, the industry generally has the following two practices, the first, that the choice of sharding algorithm for users is a mental burden, should be completely hidden from the user, another view that should give users complete freedom to choose, such as some open source software, provides more than 10 kinds of sharding algorithm. DDM believes that if the selection of the Shard field and the Shard algorithm is completely hidden, it may result in unnecessary full-table scanning, wasting resources and unable to scale linearly. Because the most understanding of the business is the users themselves. Too many sharding algorithms do bring the burden of choice, some algorithms exist mainly because of the lack of the existence of smooth expansion of the last resort. DDM has designed three standard sharding algorithms, hash, range, list, and subsequent discretionary open custom algorithms.

Can you give us a detailed introduction of these three algorithms?

1. The data distribution of the Hash:hash algorithm is more uniform, no hotspot problem, the disadvantage is that if there is a partial range of queries, full-shard scanning is required. Hash data expansion needs to migrate data, DDM has a smooth expansion function, so this block do not worry.
2. Range: Data is fragmented by numeric range or date range, and queries for the range can be parallel, but the downside is that a single range may have hot issues, such as data operations over the last one months of the date, which can be performed by only one or a few machines. Range shards do not need to migrate data when scaling up, just configure the new scope to the newly added RDS.
3. List: Enumeration shards can be seen as a special case of range, which is not mentioned here.

The design of hash algorithm?

hash algorithm design, mainly taking into account with the smooth expansion of the coordination, the use of two-level mapping rules, mainly to facilitate the control of the slot to the actual datanode mapping relationship, and the consistency hash is fixed here.

What are the unique advantages of DDM in scaling up compared to traditional solutions?

Traditional practices DBAs migrate data manually, to downtime, to affect the business, and the migration process can be error-prone. The expansion of many middleware in the industry is generally in accordance with the plan of the whole library migration, such as the original 8 sub-library, the migration is only part of the library to migrate to the new RDS, the disadvantage is that the number of shards has not increased. The DDM approach is to truly redistribute data, migrate data in slots, and ensure that the data is roughly distributed evenly after the migration is complete. The number of shards increases automatically as new RDS is added. DDM in the operation of the real automation, the implementation of a one-click migration, the migration process of switching routes, clean up data are automated, do not need to stare at the user at any time to operate. Even if an exception occurs in the migration, it is automatically rolled back to ensure consistency of the migrated data. The migration process does not block the business, only briefly interrupt the write operation when switching routes, the read operation is normal, and only affects the part of the data that is migrated to write, no other data has no effect.

(with migration flowchart)

What are the considerations for DDM in routing speed and content accuracy?

About switching routing speed, although the industry many known as millisecond, generally omit the data check, or only check the number of bars. It is well-known that the algorithm has been tested more finely. DDM believes that even if the test is sufficient, it is difficult to guarantee that 100% will not be problematic. So DDM through the design of a fast check algorithm, the content of the data validation, even if the data a little bit different, the algorithm can be verified, and fully utilize the RDS computing power to improve the speed of calibration.

In general large-scale applications, some table data is large, and some table data is small and not updated, how does DDM support different types of scenarios?

For the actual scenario that the business encounters, DDM has designed three table types: Shard table: For those tables with large data volumes, tables that need to be sliced into multiple shard libraries, so that each shard has a portion of the data, and all the shards form the complete data; single table: Relatively small amount of data, There is no requirement for join queries with other shard tables. The single-table data is saved on the default when a shard, this design can be as compatible as the single table itself complex query; Global tables: There are fewer data volumes and updates, but there is a need for joins with other shard tables. The global table holds a copy of exactly the same data on each shard, which resolves the join with the Shard table to be executed directly down to RDS.

Under distributed conditions, the primary key constraints in the original database will not be used, is it necessary to introduce external mechanisms to ensure the uniqueness of the data identity, then this global unique sequence DDM is how to ensure it?

The DDM global unique sequence, using a similar method to MySQL's auto_increment. Currently, DDM guarantees that the field is globally unique and ordered incrementally, but not guaranteed to be continuous. Currently, DDM designs 2 types of sequence mechanisms, DB and time. The sequence of the DB means is realized by the DB, it needs to pay attention to the setting of the step, the step is directly related to the performance of the sequence, and the size of the step determines the size of the batch fetch sequence. The time sequence uses timestamps and machine number generation, with the advantage that uniqueness is guaranteed without communication.

What are the advantages of DDM in operation and maintenance monitoring?

DDM: The use of traditional middleware operations need their own operation and maintenance, the general middleware focus on core functions, less consideration of operations and graphical interface operation. DDM leverages the advantages of cloud to provide a comprehensive graphical interface operation for instances, logical libraries, logical tables, and sharding algorithms. At the same time can be online view of slow SQL and other monitoring content, to facilitate the system for targeted performance tuning.

In what direction will the future DDM evolve?

The future direction of DDM for distributed transactions, enhanced distributed query capabilities, performance optimization, and so on, considering some of the characteristics of implementation if only from the middleware level of implementation will be more restrictive. DDM works with the underlying modifications of the database to provide better features to meet the business needs of the user.

Dialog DDM: Full analysis of distributed database middleware