Large data providers please do not degrade the Data warehouse system

I've found that a lot of big data providers are always trying to prove the superiority of their technology by debasing the Data Warehouse, and I have always hated this way of marketing. They always say that the Data Warehouse system is too large, expensive and inflexible, and that their technology is fast, flexible and inexpensive. In the end they will be smug and say, "Come buy our products, and we'll get you out of the Data warehouse." ”

They are always implying that you are a technology or that the solution itself is out of the question.

I admit that there are many problems with the data warehouse itself. It's not easy to design a data warehouse, but it's even more difficult to implement a data warehouse. Some of these criticisms are right-the data warehouse is long, costly and difficult to revise. But this is not to say that it has no value and should be replaced.

Business people are the key to releasing data Warehouse value

In essence, a data warehouse is not a technology or a tool. It is primarily a business process that consolidates organizational resources electronically, such as data, so it is a whole, not a loose stack of various components. Without a data warehouse, the Business executives can only act blindly, pass some error data, or make important decisions without data at all.

Although we need to use some technology to implement the Data warehouse, but the technology does not equal to the business objectives, nor from the perspective of enterprise development to look at data. Only business people can do these things. In fact, a more challenging and time-consuming task than creating a technology infrastructure is to allow business people to endorse the definition of core business entities. A data warehouse that is poorly designed or poorly performing should not be blamed on technical or technical personnel, but the problem is that executive directors do not have sufficient leadership, foresight, and patience to create generic business data dictionaries.

Data Warehouse system can provide neat data

Technically, a data warehouse is just a repository of data that stores neat, complete, and semantically consistent data collected from important applications and systems in your organization. We can use a variety of different technologies and tools to implement a data warehouse, including relational databases, master Data Management Center, and even open source large data processing architecture Hadoop and so on. Each technology has advantages that other technologies do not have, but at the same time no single technology can solve the problem independently. But the key to the problem is not the quality of the technology. The Data warehouse is actually an abstraction, a logical manifestation of some neat analytical data that the Executive Director will use to make decisions.

However, it seems that many people in the large data community advocate abandoning the data warehouse altogether. Perhaps what they really mean is no longer using traditional relational databases and business intelligence tools to store and query business data. There is no problem--we welcome this approach. New technologies always bring some benefits. But it still doesn't eliminate the need to get clean, complete, and reliable data.

Big data providers need to explain how they will increase corporate insight and provide standard reporting. It is a pity that most people ignore this demand and even think it is insignificant in the overall big data plan.

An analysis of the 3 pillars of the ecosystem

The reason for the noise that degrades the data warehouse, I think, is that the large data community magnifies the role of the data Warehouse itself. The Data warehouse is just one of several resource pools in the mature analysis ecosystem, and it is tied to Discovery/discovery and event-driven alarm systems (see Figure 1).

In short, the role of the Data Warehouse is to help business people monitor existing processes and activities, identify key trends and anomalies, based on a report and analysis environment that is designed to address a number of known issues. Although the Data Warehouse also supports some analysis functions, its purpose is not to solve new and unexpected problems. This is the work of exploring and discovering the environment-the unique function of modern large data movement. It enables influential users to use new and old data, execute complex queries, and then apply machine learning algorithms to generate new insights. At the same time, the alert environment can handle event-driven data from a large-capacity transaction or space to handle the system, and then alert the user or downstream system when the data triggers a predefined rule.

Figure 1 lacks technology. As mentioned earlier, we can implement data warehousing systems (and other environments) using a variety of technologies and tools. The choice depends primarily on the organization's legacy systems, budgets, and risk tolerance. However, regardless of the technology you decide to use, be sure to understand how it is integrated into a properly designed analytical ecosystem.

Finally, we cannot let large data supporters denigrate the Data warehouse. It plays an important role in any analytical ecosystem. A data warehouse is a vehicle for delivering enterprise data views and driving standard reports and analyses. Who can leave it?