What is ODS?

ODS is a topic-oriented, integrated, variable, and current collection of detailed data. It is used to support enterprises' demand for instant, operational, and integrated information. It is often used as a data warehouse transition and is also an option for data warehouse projects.

According to the definition of Bill. inmon, "Data Warehouse is subject-oriented, integrated, stable, and time-changing, mainly used for decision-making support of database systems"

ODS is a topic-oriented, integrated, variable, and current collection of detailed data. It is used to support enterprises' demand for instant, operational, and integrated information. It is often used as a data warehouse transition and is also an option for data warehouse projects.

This is defined in Kimball's <Data Warehouse liftcycle toolkit>

1. Integration in an operating system for current, historical, and other details queries (part of the Business System)

2. Provide current detailed data (part of the data warehouse) for decision support)

Therefore, operational data storage (ODS) is a data set used to support daily global applications of enterprises, ODS data has four basic features: topic-oriented, integrated, variable, and data current or near current. We can also see that ODS is a data storage technology between DB and DW. Compared with the application-oriented distributed dB, the Data Organization Mode in ODS and the Data Warehouse (DW) it is also subject-oriented and integrated, so the data entering ODS is also integrated and processed like the data entering the data warehouse. In addition, ODS only stores current or close to current data. If necessary, it can also perform operations such as adding, deleting, and updating data in ODS, although the data in DW is also subject-oriented and integrated, it is generally not modified. The difference between ODS and DW mainly reflects the data variability, current, stability, and aggregation.

Because ODS is still stored in common relational databases, I do not recommend that ODS store data for a long period from the perspective of performance, storage, backup and recovery, and the impact on the performance of the source database, similarly, data in ODS should not be converted, but should be kept intact with the Business Database. That is, ODS is only a backup or image of the business database. It aims to isolate the processing and decision support requirements of the Data Warehouse from the OLTP system and reduce the impact of the decision support requirements on the OLTP system.

Why do we need an ODS system? In the system architecture with ODS, ODS has the following functions:

1) form an isolation layer between the business system and the data warehouse.

Generally, data warehouse application systems have very complex data sources, which are stored in different geographical locations, different databases, and different applications, extracting data from these business systems is not easy. Therefore, ODS is used to store data directly extracted from the business system. The data is basically consistent with the business system in terms of the data structure and logical relationship between the data, therefore, the extraction process greatly reduces the complexity of data conversion, and focuses on data extraction interfaces, data size, and extraction methods.

2) transfer details query functions of some business systems

Prior to the establishment of a data warehouse, a large number of reports and analyses were directly supported by the business system. In some complicated report generation processes, the operation of the business system was under considerable pressure. ODS data is consistent with the business system in terms of granularity and organization. Therefore, the query of reports and detailed data generated by the business system can naturally be performed from ODS, this reduces the query pressure on the business system.

3) complete some functions that cannot be completed in the data warehouse.

In general, in the data warehouse architecture with ODS, the data stored in the DW layer is summarized data and operational indicators, and does not store the detailed data generated by each transaction, however, in some special applications, you may need to query the transaction details. In this case, you need to transfer the detailed data query function to ODS to complete the query, in addition, ODS data models are stored in a topic-oriented manner, which allows you to easily support query functions such as multidimensional analysis. That is, the Data Warehouse meets the decision support requirements of enterprises from a macro perspective, while the ODS layer reflects the detailed transaction data or low-Granularity Data Query requirements from a micro perspective.

In the architecture of a data warehouse application system without the ODS layer, the data granularity stored in the data warehouse is determined based on the needs, but generally, the most detailed business data needs to be retained, which is actually equivalent to ODS. However, unlike ODS, the detailed data at this time is not "Current and changing" data, it is "historical, no change" data. The Storage pressure and performance pressure of such a data warehouse are relatively large. Therefore, it puts forward higher requirements for the physical design and logic design of the data warehouse.