DM Architecture Design

Dm7 Architecture Analysis http://www.ccw.com.cn 10:27:50

Overview The dm7 database management system is based on the experiences and lessons learned from the DM series products, absorbing the advantages of mainstream database products, learning specialized database technologies adapted to OLAP applications, and adopting Java-like virtual machine technologies, A new generation of database products. The new architecture greatly improves the performance, language richness, and scalability of DM database products. It can take into account both OLTP and OLAP requests and fundamentally improve the quality of DM products.

Compared with DM6, there are hundreds of new and improved features in dm7. This article will introduce the most important and innovative features of dm7.

The core architecture of the system dm7 adopts an efficient multi-threaded architecture. Different functional threads collaborate to complete the overall functions of the server.

Figure 1 System Structure

As shown in figure 1, the database Kernel provides functions such as lexical and syntax analysis, transaction management, blocking, multi-core query engines, security management, Backup recovery, buffer, logs, and the operating system and hardware abstraction layer. On this basis, you can build two types of cluster structures based on the database kernel: Large-scale parallel clusters without sharing and clusters with shared disks. On the client side, you can access DM database or database cluster through various standard interfaces provided by dm7, or manage DM database through various client tools.

Operating system and hardware abstraction layer: Provides encapsulation APIs for the operating system and hardware environment, allowing a set of source code to run on different platforms, shielding operating system and hardware differences, and simplifying upper-layer processing logic. The storage format and message communication format are consistent in various operating systems and hardware environments.

Buffer Management: This module loads data to be accessed from the disk to the memory, and provides efficient and high-concurrency buffer access and elimination mechanisms, to support data access from a large number of concurrent users.

Log Management: Provides the persistence capability of transactions, and any committed transactions take effect permanently.

Transaction Management: provides a multi-version concurrency control mechanism to ensure the atomicity and isolation of transactions, and provides high concurrent access capabilities. The query can be blocked at any time.

Lock Management: Provides the blocking capability when concurrently updating data to avoid data inconsistency.

Multi-core query engine: It has multiple built-in Query Processing engines and uses adaptive query engines for Different queries, such as full-text search engines, object-link Data Processing engines, and Spatial Data Processing engines.

Security management: provides a built-in security mechanism to ensure that the security level of the system can reach level 4.

Backup Recovery Management: provides various Backup recovery functions, such as full backup, Incremental backup, parallel backup and recovery, and time point-based recovery.

Non-shared cluster: Based on the dm7 database, the high-performance large-scale parallel cluster function is provided based on data division. multiple nodes can process the same request in parallel to improve system performance, in addition, the inter-node daemon technology can be used to ensure uninterrupted services when any node in the system fails.

Shared disk cluster: Based on the dm7 database, the cluster function of the shared disk architecture is provided. Multiple cluster nodes share one copy of data, which can improve system performance by adding/deleting nodes, it can also provide uninterrupted services when a node fails.

JDBC/ODBC interfaces: provides common JDBC, ODBC,. NET data provider, PHP, oledb and Other interfaces in applications for various applications to call.

Management/interactive tools: Provides Graphical Database Management and SQL Execution tools to facilitate DBA/database users to directly manage databases.

Performance monitoring tool: used to monitor system performance, help database administrators analyze system bottlenecks/errors, and provide appropriate system adjustment/optimization suggestions.

Migration tool: Used to migrate data from other mainstream databases or earlier DM databases to the latest dm7 database management system.

Console tool: used to configure various parameters of the DM database and perform offline backup and recovery.

Data Warehouse suite:

Other tools: such as database replays tools to help reproduce problems, and quick data loading tools to quickly load data from text files to dm7.

High-performance Query Processing Technology dm7 adopts a cost-based Query Optimization Strategy, uses a multi-trip processing strategy, combines with Stack-Oriented Virtual Machine actuators, and uses reuse and other optimization methods to achieve optimal query results.

The optimization process of query statement analysis is not achieved overnight, but is transformed in multiple stages until the physical query plan is finally generated.

Parse: performs a syntax check to convert the original statement to the corresponding syntax tree.

Phase A: Performs Semantic Analysis and translates identifiers into dictionary objects.

Phase B: relational transformation.

1. Convert the syntax tree to the rel tree according to the rules;

2. Process subqueries and convert in and exists queries into semi join queries );

3. process the subquery relevance and remove the relevance to join the referenced table ).

Phase C: uses statistics to optimize the cost.

1. select an appropriate index for a single table query.

2. For multi-table links, select the connection sequence and the physical connection operator.

Phase E: expression Optimization

1. Reuse the same expression in the statement to avoid repeated Computation

2. Select batch commands for computing to reduce the cost of repeated calls.

Phase G: Parallel Optimization, construction of parallel operators, and multi-threaded processing of horizontal partition table queries.

The dm7 query optimizer uses optimization rules to convert all related subqueries into equivalent link connections, thus canceling the nested relationship of the query.

For example:

Select count (*)

From c_order

Where exists (

Select *

From c_order_line

Where o_id = ol_o_id

And ol_quantity> 10

)

The preceding link tree is converted into a link tree equivalent to the following statement (pseudo SQL:

Select count (*)

From c_order A semi_join (

Select B. rowid

From c_order_line, c_order B

Where B. o_id = ol_o_id and ol_quantity> 10

Group by B. rowid

) On A. rowid = B. rowid

For semi-joins or nested connections introduced due to removal of relevance, certain rules can be used for optimization. In the preceding example, the link tree can be further optimized to the following statement:

Select count (*)

From c_order semi_join c_order_line

Where a. o_id = ol_o_id and ol_quantity> 10

The flattening of related subqueries greatly reduces the complexity of Cost-optimized algorithms, making it easier for the optimizer to generate better query plans.

Based on the previous virtual machines, dm7 has made the following improvements to further improve functions and improve execution efficiency.

1. Use a standard stack with the word length as the allocation unit to improve space utilization, and make full use of the two-level CPU cache to improve performance.

2. added the stack frame concept to facilitate function/method jump and provide the basis for debugging PL/SQL scripts.

3. added the concept of memory running heap to implement object, array, and dynamic data type storage.

4. Use the stack-oriented expression computing mode to reduce the volume and data movement of virtual machine code.

5. Redefine the command system and add access to objects, methods, parameters, and stacks to facilitate PL/SQL Execution.

The structure of the virtual machine is shown in.

The dm7 stack-Oriented Virtual Machine executor can smoothly switch between commands and operators and easily call processes/functions. With a simple and efficient command system and an intuitive and fast stack data transmission mechanism, you can effectively improve the execution efficiency of SQL scripts or processes/functions.

From analysis, optimization to actual execution of SQL statements, each step consumes system resources. Reuse of query plans can reduce repeated analysis operations and effectively improve statement execution efficiency. Dm7 uses parameterized constants to make query statements with different constant values reusable. After optimization, the reuse strategy of the plan is more practical in the application system.

The query result set can also be cached. If the table data involved in the same query statement does not change, you can directly reuse the cached result set. The cache of query results improves the performance of OLAP applications that do not frequently change data, or have a large number of application environments similar to cataloguing function queries.

Implementing result set caching on the server can improve the query speed while ensuring the real-time and correctness of the cache results.

The powerful and flexible language analysis DM database provides flexible language support for database operations. DM not only supports powerful PL/SQL, but also supports BLOCK statement blocks in C languages.

1. PLSQL

PL/SQL is a procedural language for DM databases, including a complete set of data types, condition structures, cyclic structures, and Exception Handling structures. PL/SQL can execute SQL statements, PL/SQL functions can also be used in SQL statements. PL/SQL supports all SQL data types and all SQL functions. Supports cursor, % rowtype, % type, exception, and array types, and all types of DM objects. For example:

Declare

Cursor CC (x INT) is select ID, name from sysobjects where ID <X;

R CC % rowtype [];

CNT int;

K CC % rowtype;

Begin

Open CC (20 );

CNT = 10;

R = new CC % rowtype [CNT];

For I in 1 .. CNT Loop

Fetch CC into R [I];

End loop;

For I in 1 .. 9 Loop

For J in I + 1 .. 10 Loop

If R [I]. Name> r [J]. name then

K = R [I];

R [I] = R [J];

R [J] = K;

End if;

End loop;

End loop;

For I in 1 .. CNT Loop

Print R [I]. ID;

Print R [I]. Name;

End loop;

Close cc;

End;

/

2. Block statement Block

DM database provides BLOCK statement blocks in C language, supports basic data types in C language, supports condition structures, cyclic structures, exception handling structures, and goto structures in C language format, BLOCK statement blocks support SQL statements and functions. In addition, the block statement block also supports reference cursor, % rowtype, % type, exception, array type, and record type. For example:

{

Sysobjects % rowtype R;

Sysobjects % rowtype y [];

Ref cursor cc;

Int I;

Y = new sysobjects % rowtype [10];

Open CC for 'select * From sysobjects where ID <10 ';

I = 1;

Do

{

Fetch CC into R;

If (CC % found)

{

Y [I] = R;

I ++;

}

}

While (CC % found );

For (I = 1; I <= 10; I ++)

{

R = Y [I];

Print R. ID;

Print R. Name;

}

}

/

Large-scale concurrent processing technology dm7 makes full use of the multi-thread concurrency feature in transaction processing, task scheduling, data organization, and other aspects to reduce synchronization conflicts and effectively guarantee large-scale concurrent processing

The improvement of transaction processing concurrency is first reflected in the support for multi-version concurrency control. Dm7 adopts the "historical backtracking" policy to implement native support for multi-version concurrency control of data. Dm7 transformed the structure of data records and rollback records. Add the recently modified transaction ID and the corresponding rollback record address of the field record to the data record, the transaction ID and the corresponding rollback record address of the update operation on the row are also recorded in the rollback record. The link between the data record and the rollback record is used to construct the complete update history versions of a row of data.

Using Multi-version concurrency control technology, query and update operations do not interfere with each other, effectively improving the execution efficiency in High-concurrency application scenarios.

Secondly, dm7 adopts the DDL high Concurrency Technology. DDL statements are converted into basic DML operations in the analysis phase. With the help of the dictionary object localization policy, the entire data dictionary does not need to be blocked during DDL execution, this effectively reduces the impact of DDL operations on the overall system's concurrent execution.

In the dm7 transaction processing, the blocking performance is also improved by optimizing the row lock resources. Dm7 improves the row lock structure by adding bitmaps to the lock to record adjacent data rows. By default, a row lock can register up to 64 consecutive rows. In the continuous update operation, this construction can effectively reduce the number of actually generated row locks, thereby reducing the probability of lock upgrade and the side effects.

Dm7 adopts a more effective asynchronous checkpoint mechanism, which no longer occupies the system buffer for a long time. The logic is simpler and the speed is faster, which has less impact on the overall system operation.

In terms of task scheduling, dm7 uses a dynamic and adjustable working thread mechanism to automatically adapt to and adjust the actual application environment and balance the rapid response with resource control.

The number of initial worker threads of dm7 is specified by the configuration file. As the number of Session connections increases, the worker threads also increase to ensure that each session has a dedicated worker thread to process requests. This ensures that all user requests are responded in a timely manner. However, when the session connection exceeds the preset threshold, the number of worker threads does not increase. Instead, the session polling thread receives all user requests and adds them to the task queue. Wait until the worker thread is idle and extract the request task from the task queue for processing.

The execution module adopts the Parallel Processing Mechanism in the query. Dm7 supports the horizontal Partitioning technology to split a data table into multiple relatively independent sub-tables. When performing a query operation on a partitioned table, multiple subthreads can scan data in different subtables with the help of parallel operators to implement parallel processing within the query.

In terms of data organization, concurrency support is also improved. Instead of the original logic rowid, dm7 uses the actual physical address as the record ID. The physical rowid is used to remove the rowid field in the data record and reduce the data size. When loading and inserting Concurrent Data, you no longer need to wait for the rowid in order, or compete for B-tree resources for rowid sorting, thus improving the concurrent processing efficiency.

Efficient memory management uses different memory management modes based on different functions of the system's submodules.

1. Memory Pool

The application and release of small pieces of memory frequently issue system calls, which may lead to thread switching and reduce system operation efficiency. The memory pool requests a large amount of memory from the operating system at a time. When the system needs to apply for memory during operation, it applies in the shared memory pool. When the memory is used up, and then release it, that is, return it to the memory pool. The system manages the memory pool effectively, so that the application and release of small memory will not have a great impact on the system.

2. Data Buffer

To alleviate the access bottleneck of the global buffer and further improve efficiency, dm7 adopts a multi-buffer mechanism to partition the global buffer into Multiple shards. The data page uses its page number to go to the corresponding buffer shard. Each buffer Shard has its own critical section. Accessing different buffer shards does not cause access conflicts. This mechanism can reduce the conflict probability of concurrent access to the global data buffer.

3. Dictionary Buffer

Each database operation involves data dictionary information. The efficiency of accessing data dictionary information directly affects the operation efficiency. DM sets a proprietary buffer for the dictionary. To ensure the isolation of things, the execution plan in DM does not directly use the contents of the Dictionary Buffer, but generates and uses a dictionary object unique to the contents of the Dictionary Buffer.

High Availability 1. Fast Automatic fault recovery

The dm7 server is always a process. No matter what protection measures are used, it is impossible to completely avoid system faults, such as operating system crashes and power loss. After an external fault is eliminated, the DM server needs to be restarted and the damage caused by the server needs to be quickly eliminated, restored to the status before the fault, and data services should be provided to the client again. The redo log system of dm7 is a tool that helps the database system recover from faults and avoid data loss and integrity damage.

Dm7 transformed the redo log management policy. The logic lsn value is used to replace the original physical file address ing to the lsn generation mechanism, which greatly simplifies the redo log processing logic and improves the log stability and reliability. Redo logs can be compressed and stored, effectively reducing Io costs.

The parallel processing mechanism is used to execute redo logs during fault recovery, effectively reducing the time spent on redo logs.

2. Primary and standby systems based on redo logs

The master-slave system is an important means for the DM database management system to improve the disaster tolerance capability. The system is composed of one host and one or more slave servers. The host provides normal data processing services, while the standby host maintains data synchronization with the host at all times. Once the host fails, one of the slave servers can immediately switch to a new host to continue providing servers. The Master/Slave switchover is automatically completed through the server, observer, and interface, and is almost completely transparent to the client.

The primary and standby systems of dm7 are developed based on the optimized redo log system to provide more stable and reliable functions. The compressed log data is transferred between the master and slave machines, greatly improving the communication efficiency.

It is very easy to build the dm7 active/standby environment. First, generate a full backup on the host and restore it to the backup machine. Connect to the master and slave machines. The host can use archive logs to automatically synchronize data with the standby server. The construction of the master-slave system is complete. Using backup files and archive logs, you do not need to perform complex operations or wait a long time to build the master and backup environments.

The configuration mode is added to the server. The dm7 server system can be smoothly transformed between the single-host system and the master-slave system without stopping services. Its practicality in special environments is self-evident.

With the improved dictionary cache mechanism, the primary and standby dm7 systems can provide full-featured database support. The client does not have any functional limitations on accessing the host system, and the slave can also be used as a read-only image of the host, supporting read-only query requests from the client.

3. reliable backup and recovery

The DM database system provides cold/hot backup and restoration functions for the database or the entire server to protect and migrate database data.

Based on the original database-level backup/restoration function, dm7 adds the table-level physical backup function.

The table-level backup operation saves all data pages of the specified table B tree in the backup file and records its association information on the control page. During restoration, a new page is allocated one by one from the target data segment, the content is copied, and the new B tree is restored Based on the association information in the record.

Table-level physical backup combines the speed of physical backup and the pertinence of logical backup, which is highly practical. Table-level backup can be used as a solution for data migration and fast loading of specific objects. Database backup and Restoration Based on table-level backup technology can also achieve database file reorganization and contraction.

4. Logical logs

In addition to physical redo logs, dm7 adds logical logs. Logical logs record all data changes in database tables, such as insertion, deletion, and update. You can specify logical logs for some tables or set all table records. With logical logs, the dm7 system provides advanced functions such as operation analysis, data replays, and data replication.

5. Advanced Replication

The replication function of dm7 is based on logical logs. The host sends the logic log to the slave, and the slave simulates the transaction and statement to repeat the data operations of the Host Based on the log. Compared with statement-level replication, logical logs can more accurately reflect the time series changes of host data, thus reducing conflicts and improving data replication consistency.

Massive Data Analysis in data analysis applications, queries often need to continuously scan over multiple rows of data. This puts forward new requirements for fast data processing. In the dm7 system, column-store is used to accelerate the I/O loading of data. Then, batch processing of data becomes a quick tool.

1. columnar Storage

The primary difference between a column-store table and a normal row-store table is that the data of each column in the column-store table is stored together and is no longer stored in records. A batch of data is stored in one partition, with the maximum and minimum values recorded in the partition.

One of the most obvious advantages of column storage is that the selection rules in the query are defined by columns, and the maximum and minimum values of the partition data are stored in each zone description, the data of each column is continuously stored. Therefore, each column in the entire column storage table is automatically indexed.

Another advantage of using columns to store tables and organize data is that column storage is more effective than traditional relational databases in terms of compression.

Another important advantage of using columns to store tables to organize data is the amount of data read and its throughput. To read data based on columns, you only need to read the data you want to know.

2. segmented Data Compression

Traditional table data stored in normal rows cannot be compressed in a common way, mainly because of the following two problems:

1) Row-based data storage is not conducive to compression. This is because the data types of each column stored in the row are generally different. If the data is compressed together, there are not many duplicates and the compression efficiency is very low. Therefore, each column is compressed separately, however, we still find that, even so, the row-store data compression rate can be up to 5-10%;

2) In the OLTP environment, a large number of reads and updates are mixed. Each update requires compression, while reading only requires decompression. Most data compression algorithms are 4 times slower than decompression. This launch will significantly reduce the transaction processing efficiency of the OLTP database engine, making data compression expensive and almost intolerable.

In data warehouse applications, data compression can be at a much lower cost in exchange for greater benefits. This includes reducing the storage capacity requirements and increasing the data throughput, which is equivalent to reducing the query response time.

The dm7 column storage supports data compression. This is because the data in each column is stored in a row by column and the adjacent field values have the same data type. The binary value range is usually much smaller, so compression is easier, higher compression ratio. After our long-term experiments and conclusions, we can usually compress data stored by column by more than 50%. This makes the column-store tables of DM database have excellent query performance while reducing the storage space requirements.

The implementation of the dm7 column storage table is that each column is divided into several partitions, which facilitates the storage of statistics of corresponding data and improves the data query speed, more importantly, we can efficiently partition and manage the data in each column so that we do not necessarily scan the entire table while compressing the data, processing each partition as a compressed package (segmented compression mechanism), rather than compressing the entire data column (so difficult to manage and inefficient ), this segment compression effectively improves the system efficiency, because if a partition needs to be scanned, it is loaded into the memory and extracted, during subsequent processing, all data in this zone is stored in the memory, which not only reduces the I/O volume, but also improves the query speed.

3. Batch Data Processing

After the data is read into the memory, according to the traditional policy, the final result set can be generated only after row-by-row filtering, connection, computing, and other operations. In the case of massive data processing, a large number of repeated function calls and repeated data copying and computing costs are inevitable.

In dm7, the batch data processing technology is introduced, that is, reading, computing, transmitting, and generating a batch of data. Batch Data Processing has obvious advantages:

L execution of batch computing for data in close proximity to the memory can significantly increase the operating system cache hit rate and improve memory processing efficiency.

L data is extracted and transferred in batches rather than in a single row, which can significantly reduce the number of function calls that flow data between upper and lower-level operators.

L using an optimized reference method to transmit data between operators can effectively reduce the cost of data replication.

L system scalar functions support batch computing, which can further reduce the number of function calls.

After the batch data processing policy is adopted, the efficiency of batch data query can be doubled.

Dm7 Massive Parallel Processing Framework dm7's massive parallel processing (MPP) Framework provides high-end Data Warehouse solutions, it provides features such as massive data storage and processing, highly concurrent processing, cost-effective, and high availability, and provides customers with industry-leading computing performance at a very low cost.

In the MPP architecture, data storage and query are automatically parallel. The inserted data is automatically partitioned and saved to the corresponding node, and the query is executed concurrently on the database engine of each node during query, all results are automatically collected to the original query node.

The MPP architecture adopts a completely peer-to-peer non-shared architecture. Each database server is called an execution point (EP), and each EP is an independent DM database. Each EP node has the same function. You can connect to any EP node in the MPP System for data operations. Shows the overall MPP structure.

The Mal system is responsible for efficiently driving data streams between nodes related to the task during the execution of the query plan, and sending messages, moving data, and collecting operation results. Message communication and data movement between EP nodes are completed by the mal communication system embedded in the DM database system.

A message in the MAL system is equivalent to a mail delivered to the corresponding mailbox. Once a mail is received, the system immediately notifies the system to retrieve the mail from the mailbox, process the mail, and complete data integration. This efficient communication mechanism is conducive to improving the efficiency of parallel queries and giving full play to the advantages of high-speed networks.

Dm7 uses the hash-based method to automatically allocate data to each EP node, that is, when creating a table, you can specify one or more columns as the "hash distribution" key, or select the first column of the table by default. The system calculates the hash value for each inserted data row, place the data on the EP node to which it belongs.

The MPP System adds three communication operators on the basis of a single node, including collection, distribution, and broadcast. They are responsible for the master EP and slave EP, and synchronize and move data from the EP. The MPP System query plan is based on a common query plan for a single node. You can select an appropriate location to insert a communication operator node, and make full use of the query optimization mechanism of the Single Node system, generate the optimal query plan.

When a user connects to any EP operation, the execution of the Communication operator can distribute the plan to each EP and return the result after parallel processing of distribution.

You can configure the data daemon function for each EP node to enable automatic failover and recovery for node failures. This meets your data reliability and security requirements, the Database Service is interrupted for a long time due to node database instance exceptions and hardware faults, which improves the system availability and reliability.

Security dm7 further improves security at the original security level.

1. Identity Authentication

Based on the original operating system identity authentication, Kerberos-based centralized authentication, and database identity authentication, dm7 adds a PKI-based identity authentication method and supports the combination of multiple authentication mechanisms.

2. Multi-Permission Separation

Dm7 is used to transform the original three-Permission separation permission management model to implement the "division of duties" permission model based on the data owner, based on the responsibilities of each role in the application system, carefully set the system permissions that each role can possess. Compared with the traditional model, the new model transfers data control from the database administrator to the data owner, so that the Administrator and non-authorized data users cannot access and explore the data.

3. Mark and force Access Control

Dm7 expands the scope of objects to be marked based on the Level 4-Structured Protection Level requirements in the national information security technology-database management system security technical requirements, all subjects and objects can be marked by default or by system security personnel. Meanwhile, the tag granularity is further refined to the column level, supporting mandatory access control for all subjects and objects.

4. Security Audit

The audit of dm7 further improves the independence of audit data. audit files can be stored and copied from the database system, and dedicated tools are provided for maintenance operations such as retrieval and merging. Auditors can categorize audit records and store them in different audit files, improving the flexibility of Audit Configuration. Auditors can set to write audit records to multiple audit files at the same time to ensure the reliability of audit records. At the same time, dm7 also supports real-time audit analysis, real-time alarm generation, Process Termination, potential infringement analysis, exception detection and simple attack detection.

5. Encryption Engine

Dm7 implements the encryption engine, which can coordinate with the password module supporting the line level, and realize seamless and arbitrary switching between multi-thread data streams, it supports various existing stream passwords, group passwords, all their working modes, and asymmetric passwords, and solves the problem of random cross encryption of multiple cryptographic algorithms and multiple keys.

Conclusion As the most innovative product of DM so far, the dm7 Database Management System creatively integrates various cutting-edge technologies of information processing, achieving a leap from catching up with the database technology to leading the technology. The new generation of products are greatly enhanced in terms of versatility, high performance, reliability, security, scalability, data analysis, and ease of use. The implementation of its many enterprise-level features enables dm7 to fully meet the needs of large and medium-sized enterprises, as well as core business systems such as finance and telecommunications, and is an ideal enterprise-level data service platform.