Evolution and practice of cloud database architecture

Today, large enterprises such as financial enterprises and banks, in the next generation of micro-service architecture transformation requirements, the need for basic software and data platform to achieve native cloud, to meet the needs of micro-service architecture.

MicroServices, a service-oriented, loosely coupled architecture with specific boundaries.

The main features include, each micro-service is an independent autonomous system, can not rely on external components to run independently, the application only exposes the interface, the user can flexibly adjust the use of each micro-service, business granularity is small enough.

In the process of "cloud" of Enterprise Architecture , the cloud of database is the most important and difficult part. The Database Cloud Platform (DBPAAS) is a class of database management Systems (DBMS) or storage management systems that support elastic expansion, multi-tenancy, self-management, and the ability to run on top of the infrastructure (IaaS) of a cloud service provider.

according to The Gartner report predicts that the market share of the database cloud platform will double in the next five years, while 70% of users will start using the Dbpaas database cloud platform. Therefore, in order to meet the requirements of various applications for the database cloud platform, and in order to reduce the operational complexity of the large number of different types of data storage products in private cloud deployment, the database architecture evolution will be one of the main directions of database transformation in the next ten years.

Technical Requirements for cloud databases

In the process of "cloud" of business and application , the importance of cloud database in the overall architecture is self-evident in the cloud transformation. The core requirements of the cloud database are a few points, mainly:

• Elastic expansion Capacity: Database capacity needs to expand according to business elasticity, to meet the capacity requirements of different businesses;
• Flexible deployment and on-demand capability: In addition to database storage, other database functions also need to adapt to the needs of the application, flexible deployment;
• Data reliability and service continuity: reliable and secure data, full-time online is required for all business;
• Compute storage Separation: Flexible configuration of computing and storage resources, can choose a variety of computing methods can also correspond to a variety of storage methods, to meet more business needs;
• Multi-mode storage capabilities: structured, unstructured, semi-structured, and graph-like storage of multi-type data;
• Self-Management capability: Provides 0 downtime maintenance, continuous integration, and rolling upgrade capabilities to improve developer efficiency;
• Self-monitoring and problem-repair capability: fault monitoring and problem repair, reduce operation and maintenance costs;
• Whether to meet specific scenarios: pluggable components or tools for specific scenarios;
• Regulatory and security: meet regulatory requirements and ensure data security.

cloud databases need to meet these technical requirements, in addition to the specific improvements in functionality, the overall architecture needs to be upgraded and "evolved."

Cloud Database Schema Direction

Cloud Database architecture is the key point of its ability to carry out the "cloud" of application architectures, and with the development of technology and business, the architecture of cloud database has several major development directions:

• In the Dbpaas platform, the compute -storage layer separation will become the mainstream technology direction. By decoupling the protocol parsing, computing and other modules from the underlying storage, the database cloud platform shards the storage layer to achieve the elastic level expansion of storage, while the non-state design of the computational layer allows the computational layer to increase the computing power linearly by increasing the number of nodes, which has reached the elastic level expansion of the whole database cloud platform.
• multi-mode architecture becomes the mainstream trend, The Multi-model architecture can support multiple storage methods in a single database platform, greatly reducing the cost of operations and development. Traditional databases such as IBM, Oracle, and so on have provided relational, OO, and even XML storage engines. The new generation of databases provides Newsql, JSON, graphs, object storage and other types of data storage engines.
• The Cloud database platform will provide a variety of mixed-mode data Services – relational and non-relational. This mode enables users to combine different types of data storage in the same platform to provide hybrid data storage solutions for next-generation IT applications.
• more in line with the requirements of the MicroServices business architecture, microservices require that each service module be as loosely coupled as possible and independently extensible. Therefore, for the database, but also for different business, different focus on the configuration, whether it is the traditional "read-write separation" or the current popular htap is around this requirement.

In view of these main development direction, we will discuss several important technical features of cloud database in detail.

1) Storage-sql separation

A new database architecture is becoming mainstream in the requirements and architecture direction of the cloud database, which is the database The storage-sql detach schema.

The storage -sql separation architecture refers to the database's storage engine and the SQL engine, which are loosely coupled to each other and work independently of the architecture. Typically, this architecture is divided into three sections of storage, SQL, and metadata.

• Storage tier: The storage engine for the database, which handles the storage management of the data. It also includes routing and transaction control to ensure the acid characteristics of the data . In addition, the storage layer should also have indexes, query condition filtering, sorting, and a series of functions.
• SQL layer: The SQL layer is primarily responsible for processing SQL requests, and the upper layer directly faces the application, distributes the application's access requests to the storage tier, and accepts the data results returned by the storage tier.
• Metadata area: The metadata area is responsible for storing all metadata information for the entire database.

A typical cloud database architecture diagram

for this architecture, in fact The current schema of the MySQL database is somewhat similar.

The MySQL database's SQL, storage-separated architecture is more flexible in its architecture, and its open source ecosystem supports the full docking of different products, engines, and tools. On the architecture of the storage engine, the plug-in storage engine architecture separates query processing from other system tasks and the storage extraction of the data. This architecture can choose the right storage engine based on the needs of the business and the actual needs.

MySQL database overall technology module architecture

as shown, MySQL's storage engine can mount a variety of different products, each of which provides different technical features. These include InnoDB, MyISAM and other architectures.

Storage and The SQL decoupled architecture is now popular in the database industry, and AWS's Aurora database uses a similar architecture for SQL access. SEQUOIADB 3.0 is currently on MySQL compatibility, and is mainly to take "sql-storage Separation" architecture.

SEQUOIADB 3.0 MySQL compatible logic architecture

SEQUOIADB 3.0 uses a MySQL database native SQL parser that supports the MySQL protocol naturally and can do 100% syntax compatibility. In this architecture, the MySQL protocol parsing layer serves as the role of SQL parsing and distribution, directly facing the application, and each access node of the MySQL service is an independent MySQL process that supports read and write operations. The data storage and management layer is completely implemented by the distributed database engine of the giant FIR database. Simply put, SEQUOIADB 3.0, as the InnoDB replacement engine for MySQL, provides the ability to elastically expand the database storage layer while naturally supporting all of MySQL's syntax and functionality.

2) Multimode Multi-model

Enterprises using Cloud Database docking more and more applications, the need for a variety of, the traditional approach is to provide more than 10 different database products in Dbpaas to meet various needs, such a method after the system increases, the overall maintainability and data consistency management cost is high, Will affect the use of the entire system.

"Multimode" for cloud databases

In order to realize the unified management and data fusion of business data, the new database needs the ability of multi-mode (Multi-model) data management and storage. Database multi-mode Multi-model means that the same database supports multiple storage engines, which can meet the unified management needs of the application for structured, semi-structured, and unstructured data.

Typically, structured data refers specifically to the data storage structure of a form type, typical applications include traditional services such as banking core transactions, while semi-structured data is used massively in scenarios such as user portrait, IoT device log acquisition, and application clickstream analysis, while unstructured data corresponds to massive images, videos, and document processing have grown rapidly under the development of fintech.

The multi-mode data management capability enables the financial database to carry out the unified storage and management of data across departments and services, realize multi-service data fusion and support diversified financial services.

On the architecture, the multi-mode just mentioned Multi-model is also for cloud database requirements, so that the database using a set of data management system can support a variety of data types, so support a variety of business models, greatly reducing the cost of use and operation.

3) Disaster preparedness and multi-activity

For applications, developers do not want to spend a lot of time designing applications to consider the switching logic of high availability, disaster preparedness, and multi-live applications of the underlying data. In general, a mature Dbpaas layer should encapsulate as many operations as possible of the underlying data multi-copy synchronization, disaster switching, high-availability takeover, and complete transparency for the application.

In traditional application development, developers use middleware containers to configure data sources, and the underlying uses F5 or other virtual IP addresses to encapsulate multiple data sources. However, in the evolution of the cloud, the underlying database from a single node to the distributed node transition, for the upper-level applications on the one hand to minimize the application design time to the sub-database sub-table dependency, on the other hand, but also in the data node switching, or even the data center disaster takeover process to achieve transparent application of non-awareness.

SEQUOIADB 3.0 introduces a geo-live architecture that allows applications to access local databases from any access node in a read-write manner. In the process of data reading and writing, the giant FIR database can effectively control data consistency from the bottom layer, and remotely replicate data written locally in multiple regions to ensure that the data read and written by multiple sites is exactly the same.

In addition, the giant Cedar database provides the application transparent data switching and takeover mechanism, dynamically adjusts the underlying data distribution topology logic, can dynamically and effectively eliminate the nodes in the fault data center, so that other sites will continue to provide data services without perceiving.

Compared to the traditional high availability, the multi-activity is not only a greater improvement in performance and security, but also the application of hardware and software equipment in the multi-live data center, which can reduce redundancy.

Cloud Database Architecture Benefits

With technology-driven requirements, the Cloud database architecture has several key business values:

• no sub-tables are required: Previously, a kind of database distributed transformation Direction is the relational database to the Distributed architecture transformation, MySQL Sub-Library table is one of the scenarios. Today, storage--sql-separated architectures, which have been implemented natively in the data storage tier, avoid complex, lengthy "sub-database" scenarios.
• flexible support for business needs: storage, and The SQL layer enables flexible tuning of services, storage, and flexibility to support business needs.
• multiple storage engines are compatible: because The separation of SQL and storage tiers allows the underlying storage engine to support several different engines while maintaining the same SQL interface, enabling simultaneous compatibility of multiple data engines.
• fully compatible with existing applications: because The SQL layer uses the existing standard SQL parser more, so it can be fully compatible with the original application on SQL, without any application transformation input.
• data security is available: distributed storage and loosely coupled architectures, the data has a secure multi-copy, and loose coupling greatly enhances the fault tolerance of the whole system. Compared with the traditional single-point architecture, it is possible to realize the data double-live or even multi-live architecture, and meet the compliance regulatory security requirements of the "three centers", "San Five Center".

Cloud Database application Scenarios

Driven by the new architecture, cloud databases are now being implemented in a number of scenarios with landing applications.

Traditional trading Services

High-performance, high-throughput data storage and processing capabilities, acid, and security are important features in traditional, centralized transactional business. For example, in a typical banking business, a transactional database needs to process thousands of transactions per second in a database of hundreds of millions of records in order to meet peak-time online volumes. At the same time, in order to satisfy the robustness and reliability of the production system, the traditional transaction service has very clear requirements for the security, high availability of the underlying data storage, and the deployment capability of the three centers.

As a result, cloud databases need to move traditional transactional services to the cloud platform, while also providing better support to users in terms of security and regulatory compliance.

Historical Data Services

in recent years, with With the continuous development of it technology and big data, more and more enterprises keep the data as their valuable asset for a long time. This has resulted in a growing burden of historical data on some traditional applications, resulting in poor overall application performance due to the overwhelming database load. On the other hand, with the increasing demand for big data, once archived data need to be re-online to meet the requirements of online, real-time use, query and analysis, which requires the original large offline data "online." These requirements make historical data management a must.

For the historical data service, the robustness, reliability, configurable consistency policy, performance and concurrency are of great concern due to the direct access of the external application. At the same time, strong consistency and acid are not the most important points of concern compared to traditional trading services. Since some enterprises directly run some reports and self-service queries on the historical services platform, Htap's ability is also a notable feature.

The cloud database meets these requirements through a distributed architecture for scalability and performance, and manages the historical data well.

Real-time Online services

At present, the data processing, analyzing and querying system of the production business system and the backstage of most enterprises are t+1 data ETL through the way. Recently, with the rise of stream processing technology, more and more enterprises have begun to build t+0 data bus based on stream processing technology to realize real-time data docking between different business processes. For example, the user asset view can leverage streaming technology to significantly reduce the query pressure on the backend production system while providing an excellent user experience for the user's full asset view query.

For real-time online services, the level of the database is most concerned with performance, throughput, reliability, and availability. And for strong consistency, ACID, and Htap does not constitute its most important characteristics.

The data diversification and performance of the online business require more flexible and efficient support for the database of cloud architectures.

Image storage Services

Many industries in the business operations will generate a large number of paper documents, in the information processing and regulatory requirements, these paper documents need to be scanned into image files and long-term preservation. With the development of Internet technology and centralized Operation Center, a unified image management platform is widely needed in many industries.

For a typical image platform, the total amount of data stored is enormous, the unit cost of using traditional storage is very high, and the need for life cycle management is very complex. Therefore, for the increasing volume of image data, most enterprises are difficult to query, management, and the difficulty of the expansion of several major pain points.

At the same time, because the image storage service has become a part of many processes, its stability, reliability and robustness are at the same level as the core trading system. As a result, the most interesting aspects of image storage services are reliability, consistency, scalability, throughput, and multi-modeling of unstructured storage. and its acid, htap and other characteristics of the transaction is not focused on.

Summary

Cloud database is an important direction of future database development, cloud database architecture needs to be iterated with cloud requirements, and the future evolution of cloud database architecture will continue to evolve as demand changes.

Among them, the multi-mode data engine and the separation of computing storage will be the key direction of the evolution of cloud database technology.

The giant cedar will also continue to focus on the iterative evolution of the architecture, as well as more innovations in technology and architecture for cloud architectures.

Evolution and practice of cloud database architecture