Traditional databases are declining, new OLTP databases are booming, and oltp databases are declining

Traditional databases are declining, new OLTP databases are booming, and oltp databases are declining

References:

(1) OLTP Through the Looking Glass, and What We Found There

(2) The End of an unsupported tural Era, VLDB 2007

More and more programmers begin to develop mobile apps. There are still a few programmers who actually develop underlying systems. I have seen not many materials about the development of database systems in China. I have also written my own understanding of the current development of database systems into blog posts to share with you and hope to learn from each other.

Recent Development and classification of Database Systems

WithOperating SystemDevelopment tends to be stable (excluding mobile OS), and more research is focused on the development of database systems. Few people say they want to re-build an operating system, more people are doing a variety of applications on the existing OS. However, in the past 10 years, it has been a stage of rapid development of databases. A wide range of products have emerged, such as file storage databases (such as MongoDB) and column storage databases (such as Vertica ), various NewSQL databases (such as VoltDB ). All these developments are due to the rapid expansion of data volumes.Big DataThe processing performance cannot meet the requirements. People tend to develop databases targeting different application types to meet specific data processing needs. Developing database system applications on the operating system is like developing mobile apps, it is booming. As Big Data is still a hot topic, the database that provides underlying Data management services will remain one of the fastest growing fields of computers in the future.

Many people will confuse the database system with some other concepts. In fact, as a large system, databases can be divided into many categories for products on the current market:

1. Relational Database Management System (Relational DBMS), such as Oracle, SQL Server, MySQL, and PostgreSQL

2. Key-value storage, such as Redis, Memcached, and DynamoDB

3. File storage, such as MongoDB, CouchDB, and Couchbase

4. Big data storage systems, such as Cassandra, HBase, and Google's Bigtable

5. Hadoop-based data analysis systems, such as Hive, Spark, and Impala (Category 4 and Category 5 are somewhat different .)

6. query a text system, such as Solr and Elasticsearch.

In addition to the common types above, there are many other small branches, relational databases, object databases, and so on. This is not the focus of the discussion here. This article mainly discusses the first type of traditional relational database service (RDBMS ).

Different types of databases apply to different needs, and they have different similarities. As the first type of traditional relational data system, the most obvious difference with other types of databases is: A) SupportAll SQL statementsStatement, B) supportedTransactions(Transaction)ACIDAttribute. The second and third categories do not have the following features: A and B. Most of the fourth and fifth categories do not support A and B. Even if other types support A or B, they are very different from those supported by RDBMS. For A, other types of databases only support certain SQL subsets, rather than the entire SQL standard, or older SQL standards, such as SQL92 +. For B, it is not the ACID attribute that supports all transactions at the Row level, but the eventually consistency or something is a commercial promotion term, which is actually no consistency.

This does not mean that other types of databases are not good, but we enteredDiversified DatabasesDifferent databases have their own characteristics and expertise, which cannot be generalized. For example, for Consistency, the Bank's business needs to strong consistency to ensure that funds are in and out correctly, while Weibo can discard some consistency in exchange for high system throughput, the user is not very concerned about whether to view the Weibo status of a friend even if (for example, the time delay is less than 2 seconds.

Traditional relational database system systems can be roughly divided into two categories based on applications:OLTP(Online Transaction Processing) andOLAP(Online analytical processing), in which OLTP processes transactions such as concurrency and multi-thread management, OLAP is used for massive data analysis and is part of BI (Business Intelligence. Most of the first type of relational database systems include OLTP and OLAP functions, which are generic databases. The following section also focuses on OLTP databases.

Performance Analysis and bottlenecks of traditional relational databases

In recent years, there have been many analyses on the performance of traditional databases. Personally, I am optimistic about the document OLTP Through the Looking Glass, and What We Found There jointly developed by HP and MIT. Simply put, they analyzed contemporary databases and concluded that traditional relational general-purpose databases only process valid data about 10% of the time, the remaining 90% of the time is wasted on other auxiliary work: Buffer manager, Latching, Locking, logging, and Btree keys.

This is the performance icon they run the TPC-C benchmark to draw different database sections, with the percentage of instructions on the left and the percentage of CPU cycle (that is, CPU execution time) on the right. The white part is really useful for data processing, and the rest is an indispensable part of traditional databases, but it consumes a lot of resources. As shown in, Cache Management and locks, door latches and logs are both overhead of traditional relational databases.

The performance defect of traditional databases has never been mentioned on the agenda, mainly because the data size was too small in the past. With the development of the Internet in the past 10 years, especially the explosive growth of mobile applications in the past five years, the data volume is also growing. In the contemporary era, anyone who can handle Big data and find the commercial value of Big data can make money. The competition among many technology companies is the competition of data processing capabilities. This is why many NoSQL databases and NewSQL databases have emerged in the last 10 years. Earlier in the Development of NoSQL, there were many well-known systems, such as Google'sBig Table, Amazon DynamoDB, Apache HBase, Cassandra, etc. The emergence of the NewSQL system is about five or six years later than that of NoSQL.VoltDB, NuoDB, Clustrix, etc. What they have in common is to solve the big data processing performance problem. The difference is that the NewSQL system aims to solve the problem that NoSQL does not support the standard SQL language and Transaction does not fully support the ACID attribute. In other words, NewSQL is more comprehensive and compatible with traditional data than NoSQL.

Many people want to ask, why is the popular database on the market so poor? Are you all wrong? In fact, this problem is very simple. Traditional databases have been developed very early and can be traced back to the 70 or 80 s of the last century, at least 30 years ago. The actual architecture and model of this database system were determined by the overall computer hardware level and theoretical level at that time. Hardware has developed rapidly in recent years, from Disk/RAM size to price, CPU performance and Multi-core technology, compared with 30 years ago, there has been a rapid development. Moore's law has slowed down the growth of semiconductor technology over the past two years, but it is still improving. Furthermore, the reason is that the database application was very simple 30 years ago. After so many years of development, our actual data processing needs are constantly diversified, traditional databases are also constantly adding different functions, making them larger and larger.

 

Architecture of New OLTP Database

To remove the performance bottleneck of traditional databases, researchers at MIT university completely redesigned the database based on the current hardware level, instead of making a smile on the traditional database.

Contemporary New databases also pay more and more attention to distributedScale outWhile traditional databases are still improving the processing capability of a single machineScale up. For ordinary users, it is impossible to purchase expensive mainframe and database software as large enterprises have sufficient funds. If you want to back up data and achieve High Avaliability, You need to purchase and run at least one copy.

New OLTP DatabaseSolution:

Purpose of changing the Database System	New OLTP Database Technology
Remove logging overhead	Use new logging
Removes locking, latching, and other overhead.	Data Partition+Single-thread execution
Remove buffer manager overhead	UseMemory, Replace disk read/write

According to the research results of relevant scholars, after removing these significant overhead, the OLTP relational database Transaction throughput increases at least20x.

What is OLTP in the database?

On-Line Transaction Processing online Transaction Processing system (OLTP)
It is also known as a transaction-oriented processing system. Its basic feature is that the customer's raw data can be immediately transmitted to the Computing Center for processing and the processing results can be provided in a short time. The biggest advantage of this is that it can process the input data in real time and respond in a timely manner. It is also called a Real time System ). An important performance indicator for measuring the online transaction processing system is system performance, embodied in the real-Time Response Time (Response Time), that is, after the user sends data to the terminal, the time required by the computer to reply to this request.

What is the usage of OLTP for relational databases? Do not post concepts,

Basically, the final result of OLTP is that the design of the data table is based on transactions, so it is generally highly normalized.