Five stages of the evolution of Dynamic Data Warehouse

Chief Technical Officer of--ncrteradata Division Stephenbrobst
Joerarey, chief consultant, strategic technology and systems company

The most successful data warehouses are progressively developed in a gradual manner, each of which improves the business value of information. In recent years, data warehousing has evolved to support enterprise decisions and even new heights for enterprise partners and customers.
Previously, data warehouses provided strategic decision-making capabilities in certain areas of the enterprise, such as marketing, strategic planning, and finance. The information provided by the data warehouse greatly improves the decision-making quality of these departments. However, in today's highly competitive business environment, a good strategy is just one of many elements of success. If not put into effect, any strategy will be dead letter.
The new generation of Data Warehouse application not only improves the formation of the enterprise strategy, but also develops the strategy's executive decision-making ability. This paper discusses five development and evolution stages of data Warehouse, which is also the five stages of enterprise internal decision support.

1th Stage: report
The initial data warehouse is used primarily for reporting in a department within an enterprise. The Data Warehouse integrates information from different sources within an organization into a single warehouse, which can provide an important reference for the company's cross-functional or cross-product decisions. In most cases, there has been a prior knowledge of the issues covered in the report. Therefore, the structure of the database can be optimized according to the requirements of the problem, even if the data query personnel require access to a very large amount of information, the efficiency of processing the data can still be very high.
The biggest challenge in building the first phase of data warehousing is data integration. Traditional computing environments often have hundreds of data sources, each with unique definition criteria and basic implementation techniques. It is challenging to build consistent data repositories for the cleaning of data that is not consistent across production systems.
The optimized integration information established in this stage is used for decision-makers, and also lays a foundation for the future development of Data Warehouse.
2nd Stage: Analysis
In the second phase of data warehousing applications, the focus of decision-makers shifted-from "what happened" to "why". The purpose of the analysis activity is to understand the meaning of the report data and to analyze the data in more detail at various angles. The second stage of the Data warehouse on the database to submit the problem in advance, the use of the method is mainly random analysis. The performance management relies on the advanced optimization function of relational database management system (RDBMS), because it is different from the pure report environment, the structure relation of information query is unpredictable.
In the second stage of Data Warehouse application, the performance problem is very important because the application of information base is highly interactive. Reports are typically provided on a regular basis based on the business schedule, and stochastic analysis is essentially an iterative and continuous optimization of the problem in an interactive environment. Business users want direct access to the data warehouse through a graphical user interface (GUI), and do not want programmers as intermediaries. Supports concurrent queries for data warehouses and large-volume users, which is a typical feature of the second-stage application.
Business users are often impatient, so an online analytical processing (OLAP) environment must be established, and the response time to drill down is calculated in seconds or minutes. The use of indexes and complex table join techniques allows the database optimizer to find efficient access paths. Therefore, optimizer technology is critical to the flexibility of accessing information within an acceptable response time.
3rd Stage: Forecast
When a company's decision-making process is quantified, it experiences the dynamics of the business and why the situation occurs, and the next step is to use the information for prediction. It is clear that mastering the company's impending movements means more active management and implementation of the company's strategy. The third stage of data Warehouse development is to provide data acquisition tools to create predictive models using historical data.
There are few end users for advanced analysis using predictive models, but the workload of modeling and benchmarking is enormous. In general, modeling requires hundreds of complex methods to measure hundreds of thousands of (or more) of observation data to form a predictive algorithm suitable for a specific set of business objectives. The reviews are also often used for large amounts (millions of) of observational data, because the overall evaluation is required, rather than the small amount of data used to model the modeling.
In order to obtain the desired predictive characteristics, advanced data analysis typically applies complex mathematical functions such as logarithms, exponents, trigonometric functions, and complex statistical functions. It is important to obtain detailed data for the predictive effect of the algorithm. Some tools, such as SAS and Quadstone, provide a framework for developing complex models, but they require direct access to the information stored in the Data warehouse relational structure. In the face of such applications, you must consider the capabilities of the Data warehouse. A few users can easily consume 50% or more of their resources on the Data Warehouse platform at peak times. Resource consumption is so huge because of the complexity of the data access process and the large amount of data processing.
4th stage: Operational orientation
The 4th phase of the evolution of the Data warehouse is the Dynamic Data Warehouse. The 1th to 3rd phase of data warehousing focused on supporting strategic decision-making within the enterprise, while the 4th phase was based on tactical decision support. Data warehousing support for strategic decision-making provides the information necessary for long-term decision-making, including market segmentation, product (category) management strategies, profitability analysis, forecasts, and other information. Tactical decision Support focuses on the outside of the enterprise and provides support to employees who implement the company's strategy.
In general, the "operation" of a data warehouse refers to the information that is provided at the time of the site, such as timely inventory replenishment, scheduling of parcel deliveries, route selection, etc. Many retailers tend to manage inventories from suppliers, owning a retail chain and a multitude of partner suppliers, with the aim of reducing inventory costs through more efficient supply chain management. In order for such cooperation to be successful, he must provide the supplier with the right to know details about sales, promotions, inventory in the warehouse, and then establish and implement effective production and delivery plans based on the requirements of each store and each item. To ensure that information is truly valuable, you must refresh the information at any time and respond very quickly to queries.
Taking freight as an example, the overall arrangement of freight vehicles and transport routes requires very complex decisions. It is often necessary to transfer some of the goods from one truck to another, that is, to re-load them in order to reach their destinations with the highest overall efficiency. When some trucks are late, they have to make tough decisions: whether to let the successor truck wait for the late cargo, or let it start on time. If the subsequent vehicle departs on time and does not wait for a late package, the service level for late packages will be greatly compromised. On the other hand, waiting for a late package will damage the service level of other packages to be shipped on the subsequent transporter.
The length of time the transporter waits depends on the service level of all delayed goods to be unloaded to the vehicle and the service level of the goods already loaded into the vehicle. Obviously, the next day should arrive at the destination of goods and a few days to arrive at the destination of goods, the service level and the implementation of the difficulty is very different. In addition, shippers and shippers are also important factors in decision-making considerations. For the enterprise profit is very important to customers, the service level of their goods should be increased accordingly, in order to avoid the goods late damage to the relationship between the two. The transport routes, weather conditions and many other factors of the delayed shipment should also be taken into account. Being able to make informed decisions in this situation is equivalent to solving a very complex optimization problem.
It is obvious that the LTL manager should improve the decision quality of the plan and route selection effectively with the help of the advanced decision support function. More importantly, to achieve the decision-making function of the data Warehouse, information as the basis of decision must be kept up to date. Sayin order to make the decision function of the data Warehouse truly serve the daily business, it is necessary to continuously acquire the data and populate it into the Data warehouse. Strategic decisions can use data that is updated on a monthly or weekly basis, but data that is updated at this frequency cannot support tactical decisions. At this point, the query response time must be measured in seconds, in order to meet the job site decision-making needs.
5th stage: Dynamic
The more important the role of Dynamic Data Warehouse in Decision support field, the higher the enterprise's enthusiasm for decision automation. When the effect of manual operation is not obvious, in order to seek the validity and continuity of decision-making, enterprises tend to take automatic decision. In the e-commerce model, facing the customer and the website interaction, the enterprise can only choose the automatic decision. Interactive Customer Relationship Management (CRM) used in Web sites or ATM systems is a customer relationship optimization decision-making process that is personalized in all aspects of product provisioning, pricing, and content delivery. This complex process occurs automatically in the absence of intervention, and the response time is measured in seconds or milliseconds.
With the progress of technology,more and more decisions are triggered by events and then automatically occur。 For example, the retail industry is facing a technological breakthrough in electronic shelf labeling. The advent of the technology has abolished old, hand-replaced labels that have been used for a long time. Electronic tags can be remotely controlled by the computer to change the price, without any manual operation. Electronic shelf label technology, combined with Dynamic Data Warehouse, can help enterprises to realize price management automation according to their own wishes; for seasonal goods with too large inventory, the two technologies will automatically implement a price reduction strategy to sell the most inventory at the lowest marginal loss. Price-cutting decision-making in the era of manual pricing is a very complex operation, often expensive, more than the ability of enterprises to bear. Electronic shelf labels with promotional information and dynamic pricing features bring a whole new world to price management. In addition, Dynamic Data warehousing allows users to use event triggering and complex decision support capabilities to make decisions on a per-item, per-store basis, in the best possible scenario. In a CRM environment, it is possible to make decisions based on the circumstances of each customer using a Dynamic Data warehouse.
The fierce competitive situation and the rapid technological innovation have promoted the progress of decision-making technology.Dynamic Data Warehousing can provide information and decision support for the entire enterprise, not just the strategic decision-making process. However, tactical decision support is not a substitute for strategic decision support. Specifically, dynamic Data warehouses support both of these approaches. With operational guidance and event-triggering decision support for the 4th and 5th stages of data warehousing, our strategy, developed in the 1th to 3rd phase in accordance with traditional data warehouse analysis, can be implemented.
Conclusion
The application of Dynamic Data Warehouse is a process of gradual evolution. We do not advocate jumping straight from stage 1th to stage 5th. Effective risk management should be based on traditional data warehouse applications that form a single source of data consolidation. When the data warehouse advances to have the strategic decision support function, it will inevitably put forward the higher request of the tactical decision. If the Dynamic Data Warehouse can be used for the entire enterprise, its commercial value will be greatly increased. Providing information to tens of thousands of decision-makers throughout the enterprise, even through CRM applications, to which customers can participate, will bring enormous advantages to business development. However, this will require more advanced Data Warehouse construction solutions. A data warehouse with scalable, high-performance, high-availability, and fast data-refresh capabilities that can reach top-notch service levels is not far from us.