Database Library sub-table (sharding) series (iii) Considerations of sharding Process implementation level

When the team of the system business and the database has been carefully combed, determined the segmentation scheme, the next problem is how to achieve the segmentation scheme, at present, there are many open source framework and products for reference in Sharding, at the same time many teams will choose to self-development implementation, regardless of the choice of framework or self-development, Will face a question on which level to implement sharding logic, this article will be a series of questions to analyze and consider. This text connection: http://blog.csdn.net/bluishglc/article/details/7766508 reprint Please specify the source!

First, the realization level architecture design of sharding logic

From the program architecture level of a system, sharding logic can access the wrapper layer (various spring template) in the DAO layer, the JDBC API layer, the spring data between DAO and JDBC and a sharding proxy server between the application server and the database is implemented at four levels.

Figure 1. Sharding implementation level and related frameworks/products

• Implemented in the DAO layer

When the team decides to implement Sharding on its own, the DAO layer may be the preferred location for embedding sharding logic, because at this level, each DAO method explicitly knows the data tables and query parameters that need to be accessed, which can be directly targeted to the target shard. Instead of having to parse SQL as the framework does, and then route it according to the configured rules. Another advantage is that it is not constrained by the ORM framework. Since most applications now rely on an ORM framework on the data access layer, most shrading frameworks often fail to support or support only one ORM framework, which makes it very restrictive when choosing and applying a framework, and self-fulfilling sharding has no such problem at all. Even different shard use different ORM frameworks to coordinate work together. For example, most Java applications now use Hibernate, but there is no very satisfying hibernate-based sharding framework in the present, (about Hibernate hards is described below), so many teams choose to implement sharding themselves.

A simple summary, the DAO layer self-realization of the advantages of sharding: not by the ORM framework constraints, easy to implement, easy to customize according to the characteristics of the system, no SQL parsing and routing rules matching, performance will be slightly better; The disadvantage is: there is a certain technical threshold, Workload is larger than relying on framework implementations (in turn, the framework will have learning costs), is not generic, and can only work in a particular system. Of course, the DAO layer can also pull sharding logic out of the "outside" through XML configuration or annotations to form a common set of frameworks. However, there is no such framework.

• Implemented in the ORM framework layer

In the ORM framework layer, the implementation of Sharding has two directions, one is to provide sharding support while implementing O-R mapping, thus locating as a distributed data access framework, This type of framework represents the Guzz another direction is to add the sharding mechanism by modifying the existing ORM framework. This type of representative product is Hibernate shard. It should be said that in the mainstream of hibernate, the industry's demand for a sharding framework for hibernate is very urgent, but in terms of the current hibernate shards, performance is not satisfactory, It is mainly because it has too many restrictions on using hibernate, such as its support for HQL is very limited. In terms of mybatis, there is no mature relevant framework to produce. Some people put forward the use of MyBatis plug-in mechanism to achieve sharding, but unfortunately, MyBatis plug-in mechanism to control the connection layer of multiple data sources, on the other hand, leaving the plug-in layer has lost the focus on SQL to parse and route the opportunity, so on the MyBatis framework, There is no framework available for reference, and the team may want to work on the DAO layer or the Spring template class.

• Implemented at the JDBC API layer

The JDBC API layer is a great place for many people to realize sharding, and if we can provide a JDBC API implementation that implements the sharding logic, then sharding is completely transparent to the entire application, This implementation can be directly used as a generic sharding product. But the technical thresholds and workloads of such schemes are clearly not what the general team can do, so there is basically no team to implement sharding at this level, or even such open source products. The author knows only a commercial product dbshards to adopt this program.

• Implementation of the Spring data Access encapsulation layer between DAO and JDBC

Today in Springd, there are few applications built on the Java platform that do not use spring, between DAO and JDBC, Spring provides a variety of template to manage the creation and release of resources and synchronization with transactions. Most spring-based applications use the template class as a gateway to data access, which gives us another chance to embed sharding logic. is to do sharding work by providing a template class that embeds the sharding logic. This scheme is basically consistent with the implementation of the JDBC API-based scheme, which is also transparent to the upper code, and can be smoothed over when the sharding is modified. However, its implementation is simpler than that based on the JDBC API, so it has become the choice of many frameworks, and the open source Cobar client of Ali Group Research Institute is an implementation of this kind of solution.

• Implemented by proxy between application server and database

In the application server and the database to join a proxy, the application of data sent to the data request through the proxy, the agent will be based on the configured routing rules, SQL parsing to the target Shard, because this scheme is completely transparent to the application, good versatility, So it becomes the choice of many sharding products. In this respect the more well-known product is the official MySQL agent tool: MySQL proxy and a product developed by the people: amoeba. The MySQL proxy itself does not implement any sharding logic, it just serves as a proxy for MySQL database, providing developers with a place to embed sharding logic, which uses LUA as a programming language, which is a problem for many teams to consider. Amoeba is a dedicated agent for the read-write separation and sharding, it is very simple to use, does not use any programming language, only needs to be configured through XML. However, amoeba does not support transactions (when a request from an application that contains transactional information arrives at amoeba, the transaction information is erased, so even a single point of data access will not have a transaction) has been a problem. Of course, this depends on the positioning and design of the product, and we can only say that amoeba is not suitable for systems that require very high-level transactions.

Database Library sub-table (sharding) series (iii) Considerations of sharding Process implementation level