Data Management DMS Service Platform Interprets DevOps Best Practices in Enterprise Databases

DevOps on the cloud to implement pain points

First, we will introduce high-frequency iterations for R&D. It may be very difficult for some enterprises that are not in the initial stage of cloudization or cloudization to understand why the efficiency of research and development after the cloud is high, because of the IT infrastructure on the cloud. The reduction in deployment and implementation costs is reduced, which in turn leads developers to be free from the underlying cost and capacity constraints, which puts more effort into the logic implementation of the code and the optimization of the code business architecture.

From the data point of view, the business will thoroughly practice our micro-services, that is, the business dimension is more and more subdivided, but each segment of the business dimension will have its own database, when the database business is concurrently iterative, The pressure brought by the off layer of the database is huge. 67% of the respondents indicated that the iteration cycle of the application version was greatly shortened after the cloud; the second is the sharing of data resources, introducing the concept of introducing a data asset when sharing data resources. We all know Every asset of an enterprise needs to have a responsible person who guarantees that the assets are not used indiscriminately. It is the same in the database. Each of our databases and even every record needs to have a responsible person. The data service dimension has increased significantly, and then the data coupling degree will increase. As the coupling degree increases, the responsibility boundary will become more and more blurred, so the enterprise will go to two extremes, 93% of the cloud database runs in a multi-tenant environment; The elastic expansion of capacity, the low cost of IT resources on the cloud largely benefit from the resources on the cloud, and can pass capacity bombs. Shrinking to achieve, in Alibaba's 2017 double 11 singles day, our domestic trading unit used cloud resources to sharpen and expand at the peak, and after the spike, the expansion resources will be returned to the cloud, but in the actual operation, we really want What can we do to get the resources to the cloud when we need it, and go smoothly when we don't need it.

After the pain point, we have to summarize the pain points and refine the demand. The high-frequency R&D iteration is essentially to solve the problem of quantity. The idea of solving the quantity problem is large-scale deployment, large-scale language and large-scale operation. Standardization, but standardization can not solve the characteristics of DevOps agile. The solution to sharing is to isolate the data. There are many ways to isolate the data. The essence of the container's elastic scaling is the size of the resource specification multiplied by the time occupied by the resource. How to make the resources on the cloud take the shortest time, to ensure that there is a rapid deployment of business requirements, to minimize the deployment cost of IT assets, and the ability of resource accounting on the cloud is the size of resource specifications. The hierarchical management is divided into DMS data management layer and HDM hybrid cloud database management layer. DMS data management layer can easily build the enterprise-specific database DevOps, which promotes database R&D self-help and improves R&D efficiency; HDM hybrid cloud database management provides The ability to manage multiple environments, rapid flexibility, and disaster recovery.

DevOps productization solution

The figure above is a schematic diagram of the functional matrix of the database DevOps solution. From the top down, the data operation portal is a prerequisite for implementing management and control from a technical point of view. From the business aspect, the R&D personnel's access to the database is converged. When the permission convergence is completed, the illegal account of the individual is recovered in the underlying database. The recovery of rights is always a necessary condition for data security. The security, efficiency improvement and performance analysis in the three big boxes are the three principles that DMS needs to follow. At the bottom, you can see HDM. HDM has the function of elastic migration.

In the past nine years, we have accumulated a lot of experience in the field of DevOps. We have already accumulated some experience into integrated productization. Alibaba also hopes to help you solve some problems through expert consultation and empowerment.

Best Practices for DevOps Landing

Optimizing R&D efficiency as much as possible while ensuring data and database security is the best practice of database layer DevOps. For DMS and HDM products, their responsibility is to give security and efficiency as much options as possible. The responsibility of the solution architect is to assign an optimal value to each option given by the product according to the needs of the enterprise, so as to achieve a balance between safety and efficiency in the enterprise.

The above figure is the basic flow chart of data operation by users through DMS. Data operations include query, correction and export. It can be seen from the figure that a user enters the DMS basic privilege isolation system through SQL. The first layer is to perform network access isolation, the second layer is library table privilege isolation, the third layer is field sensitive isolation, and the fourth layer is operational privilege isolation. DMS also involves enhanced privilege isolation and secure access system. DMS will set the daily query record limit and single query record limit. When these two checks are passed, in order to ensure the security of the database, DMS will check according to the predetermined rules. Query time.

The figure above shows the DMS post performance analysis architecture diagram. Through the full SQL analysis, the computing layer can make a relatively accurate trend forecast for the performance and storage capacity of the database. The computing platform discovers the problem mainly through the expert experience to diagnose and optimize the productization, and through the machine learning to carry out rapid model training and automatic discovery. And deal with online issues.

The figure above is a schematic diagram of the HCM database spike expansion solution. The cloud-like flag in the middle is DTS, which is a parallel high-speed replica that is basically independent of the data type. It evaluates the local ID data and transmits it to the cloud, and maintains the cloud data on the cloud through two-way replication. More live, HDM self-construction to the unified management of resources on the cloud and under the cloud, which is based on the overall structure of HDM elastic capacity.

Review of the solution and best practices

DMS helps us control the security of data access for employees; DMS also commercializes the database development process to help you build the ability to get online quickly; DMS post-production integration module helps database to continuously integrate and optimize; HDM achieves flexible expansion. The data management DMS enterprise is based on the accumulation of IDB in the Alibaba Group's self-developed database service platform IDB for nearly ten years. IDB serves the database management appeal of tens of thousands of people per week in the group, and precipitates the database management experience of the core DBA of Alibaba. Provide a safe and efficient full self-service database service platform for research and development.

At present, the DMS Group version-IDB has accumulated management of various business groups such as Alibaba Group, Ant Financial Service, Rookie Network, and Box Ma Xiansheng. The daily change operation of more than 30,000 database instances has gone through six years and 11 years without any production failure. At the same time, the international version of DMS is also accumulating overseas users such as AliExpress and LAZADA. DMS is divided into personal version, enterprise version, mobile version, and enterprise mobile version. Recognized by many industry users on the cloud, there are more than 10,000 service developers involved in government affairs, taxation, public security, aviation, financial securities, medical care, e-commerce, games, entertainment, etc., and there are abundant community resources and users still maintain rapid growth. trend.