Data access of Distributed system architecture design

The data access layer is a chained process, and most components provide an external JDBC implementation that provides services in three ways:

1) Provide proprietary API, poor versatility

2) Versatile way, such as JDBC

3) An ORM-based approach that can be layered on an ORM framework, exposing the interface of the original framework.

The JDBC advantage is obvious when merging queries.

Basic steps for data access layer design: SQL parsing, rule handling, SQL overwrite, data source selection, SQL execution, and result set return merge processing.

1) The parsed cache can increase parsing speed when SQL parsing is performed. An important thing is to get the table to be manipulated according to the SQL executed, and to determine the number of connections to the target data source based on the parameters and rules.

2) Rule processing, you can use fixed hash, according to a field to model, and then dispersed to different libraries and tables, usually put the periodic data together, you can also use a consistent hash, the corresponding hash value of the node into a range, is no longer discrete You can also solve the problem of increasing or decreasing the load imbalance in a node by matching multiple virtual nodes on one physical node and distributing the virtual nodes evenly. The mapping table is a method of determining the data source based on the tabular method of the value of the sub-Library's table field, which is generally used for special processing of hotspot data, or to supplement the rules that do not fully conform to the rule.

3) SQL modifications are generally used for table names, indexes, and cross-library calculations.

4) Rule processing is to determine a set of data sources, and the data source selection is the determination of a specific data source.

5) The SQL execution and result processing parts need to be aware of the abnormal judgment and processing.

In engineering, it is possible to extract driverclassname,username and password from spring into a public configuration, but preferably in configuration Management Center. Groupdatasource solves the choice problem of the specific Access database, the specific selection strategy is the key point of the groupdatasource completion, including the choice of main preparation, the choice between different libraries according to the transaction, reading and writing. Groupdatasource means that the grouping number of bindings, although not overall scaling, but can be in-group scaling, primary and standby switching, and so on. After the data source is grouped, you can build an atomic data source by slicing the data source, and the atomic data source only manages a specific database.

The deployment of the data access layer can also be divided into jar package and proxy two ways, similar to the service framework, when using proxy, the protocol can also use the database protocol and private Protocol, preferably directly using the database protocol.

Database read and write separation is best based on database log to replicate data. Data change scenarios include: Copy database, index construction, cache invalidation, etc., generally introducing data extractor to add data source change information to the data distribution platform, the data adoption device applies these changes to the corresponding target, and the data distribution platform is composed of multiple pipelines. Different sources of data change require different extractor to parse and change the work into the distribution platform. When the data is migrated, the log of the increment is recorded, and after the migration, the incremental changes are processed.

Data access of Distributed system architecture design