Large data analysis and high speed data update

Large data analysis and high speed data update

Chen Shimin

The main challenges of large data for the data management system platform can be summed up as volume (large data volume), velocity (data generation, acquisition and update speed) and produced (data variety) 3 aspects. For large data analysis systems, try to understand the importance of velocity and how to deal with the challenge of velocity. First, compare things processing, data flow, Different requirements for velocity with the data analysis system. Then from the perspective of the relationship between data update and large data analysis system, two recent research work are discussed: 1 MaSM, Support Online data update in Data Warehouse system, 2 logkv, High-speed incoming log data and efficient connection operations based on time windows are supported in the log processing system. Through analysis and comparison, it is found that the storage data update is only the most basic requirement, and more importantly, we should take the data from the update to the analysis as the whole life cycle, carry on comprehensive consideration and optimization, according to the characteristics of large data analysis, optimize the data organization and data distribution of high speed data update In order to ensure even improve the efficiency of data analysis operations.

Large data analysis and high speed data update