Sybase VLDB (massive database system) solution (1)

As the market and business models continue to refine and the system functions are increasingly enhanced, application operations will generate more production data than ever before, and enterprises will also need to analyze historical data, this will inevitably lead to extreme expansion of business data. As we can see, more and more data systems are generating hundreds of gigabytes or even terabytes of capacity. To build such a huge database application system, you need to examine it from many aspects, including the absolute performance of the system, ROI, system operation risks, data manageability, and other factors, to obtain an optimal integrated solution.

Currently, many database manufacturers support terabytes of data. However, the support for VLDB not only refers to the capacity in the database-VLDB does not simply mean that all data is stored in one table. The performance of VLDB will involve all aspects of database management: daily management, data loading, index creation, and running performance. It also needs to support a large number of user connections and large workloads. Only by maintaining good running performance can a super large database be formed. That is, any successful solution for VLDB should be or must be a complete solution.

Analysis of VLDB Processing Methods

Currently, most VLDB solutions solve the problems we face from the perspective of data storage optimization, focus less on other technical factors that affect VLDB performance, such as daily management, disaster recovery, and system availability.

From the perspective of the history of data storage technology, the entire development process of data storage is a process of centralization-segmentation-centralization-segmentation. This process is also a process that constantly boosts the development of data storage technology.

Some common VLDB storage technologies are summarized as follows:

◆Table sharding Technology
Table sharding technology is the first VLDB data storage technology. As early as the dBase and Foxbase era, people began to use the table sharding technology to make up for low PC performance and a single table with a large data volume, performance defects.

◆Table Partitioning technology
Table Partitioning technology divides a table into multiple partitions according to certain conditions and requirements. Each partition is a logical entity and a subset of the table, you can operate on table partitions like a single table, and determine the Data Management Mode of partitions based on the actual needs of application operations, for example, you can create different indexing mechanisms on the corresponding partitions for different operations, and select different parallel methods. Table partitions provide a more effective way for the system to organize data and provide the foundation for VLDB and parallel processing. Table Partitioning is a VLDB management and storage technology widely used by most database manufacturers.

◆Look-through technology
Look-through is a VLDB data storage and management method, which effectively integrates distributed technology and table sharding technology. By creating views or proxies for other database systems in the local database, it is possible for an application to Access Multiple Remote complex databases by accessing a local small database.

The Look-through technology greatly reduces the difficulty of system development and management. For developers, it only processes a logical database, and the physical location of the database is transparent to developers. It also reduces the complexity of single-point VLDB data management and hardware and software overhead. At the same time, both Table Partitioning and table sharding technologies are built on a single database system of the same type, while Look-through provides the possibility of VLDB for heterogeneous databases.

◆Application Segmentation
Whether it is table sharding, Table Partitioning, or Look-through technology, it adopts a data segmentation approach. In the real world, there are also many systems that use data segmentation while combining the application segmentation method. Establish a business system by dividing applications into function modules with relatively independent functions. Internal Services between systems can be implemented through distributed transaction processing or application integration.

Compared with other technologies, application segmentation is more like a design scheme, idea, and experience than a technology. However, due to the actual needs of applications, currently, most systems adopt the concept of application segmentation.

Through the above analysis, we can see that both the historical and existing VLDB data storage solutions have their own specific technical background, characteristics and application market. In addition, various technologies do not conflict with each other. We should make full and reasonable use of all storage and optimization technologies to ensure the efficient operation of the VLDB system.

Sybase's VLDB Solution

Sybase, as a major database software supplier, has extensive experience in establishing and implementing VLDB systems worldwide and covers many industries such as finance, telecommunications, transportation, and manufacturing. Based on user requirements and VLDB requirements for database systems, Sybase's VLDB solution consists of three parts: VLDB database storage technology, VLDB database performance optimization, and VLDB data maintenance.

1. VLDB database Storage Technology

Based on different business needs, Sybase mainly divides data storage into two modes: analyticdb data storage and transactional VLDB data storage, users can use different methods for different needs.

◆ Data storage for transactional VLDB

It is self-evident that there is an inherent conflict between the transaction speed and VLDB. High concurrency and high transaction volume are often the most direct reasons for VLDB. However, due to the data storage and indexing features of relational databases, the larger the data base, the longer the index search and data search times are, the lower the transaction speed.

The Sybase database management system integrates a variety of technologies and supports the use of table sharding, Table Partitioning, And Look-through techniques to solve the problem of managing large data volumes, at the same time, the application system can also use the application segmentation method on the Sybase Database to solve the VLDB management problem.

◆Analyticdb Data Storage

Analyticdb and transactional systems have different characteristics. Analyticdb usually focuses on one or more fact data, such as product sales and product system defective rates. Therefore, you need to analyze the long-term business status. Sybase's data vertical partitioning technology is designed for these features of analyticdb. In the management structure of Vertical Data partitions, different data types are stored to ensure fast and efficient data access and greatly improve the business system performance in the VLDB environment.

Based on vertical data partitioning, Sybase introduces patented bitwise Indexing Technology in analyticdb. Sybase's bitwise indexing technology includes regular bitmap indexing, and is specially expanded. Based on Different business data types, multiple indexing technologies are implemented, such as Fast Projection and Low Fast, high Group and High Non-Group. The most significant feature of Bitwise index is that it separates and stores data by bit based on the Vertical Data Partition, this greatly accelerates data locating and access, and also provides the foundation for data compression to control the data expansion speed in the VLDB business environment.