Data warehousing and Hadoop need to cooperate

The big data craze has sparked a lot of interest in Hadoop, as well as a misconception that Hadoop can help resolve data processing and http://www.aliyun.com/zixun/aggregation/11009.html "> Analyze the problem, it can replace the traditional data warehouse.

"Data warehouses (databases) and Hadoop (MapReduce) are two very different technologies, not a competitive relationship, but a cooperative relationship." Large data or the rise of Hadoop will not bring down the Data warehouse or database market. At the recent Sybase IQ15.4 press conference, Lu Dongming, the technical director of Sybase China, clarified this. Sybase IQ15.4 is a data warehouse product of Sybase, and one of the few data warehouses in the industry that really uses the column-type storage technology, because it uses unique column storage to make it have high compression and high speed processing capability.

Lu Dongming told our correspondent, Hadoop or MapReduce was originally to solve the problem of information search, however, the search problem and data warehouse or database problem is two completely different problems, the solution is very different ideas. For example, database and data Warehouse pay attention to the consistency of data, the same query results must be unique, and Hadoop can allow the existence of different results; The distributed processing architecture of Hadoop supports large-scale clustering, so it is easy to handle PB-level data, while data warehouses and databases are subject to many technical limitations. In addition, there is a magnitude gap in the scale of processing data.

"The Data Warehouse (or database) has a completely different application scenario with Hadoop, which is adapted to different data sizes." Instead of replacing each other, they must cooperate with each other. "Lu Dongming said.

In fact, with the advent of the large data age, more and more traditional data warehouses have chosen to work with Hadoop to meet the needs of the user's data analysis. A typical way to collaborate is to preprocess large data through Hadoop, then import the preprocessed data into the data warehouse through the tools provided by the traditional data warehouse engine, and use the data analysis and mining engine in the traditional data warehouse to analyze the data. Sybase IQ 15.4 supports this approach, and Sybase IQ 15.4 introduces many interfaces. For example, data from both datasets can be accessed simultaneously through the database's outreach interface, one from IQ and the other from Hadoop; Sybase IQ 15.4 also provides an access interface for Hadoop, so you can use a standard SQL to access the data in Hadoop via IQ.