Research on fast query algorithm of massive data

Research on fast query algorithm of massive data

Nanjing University of Posts and telecommunications Zeng She

Firstly, this paper introduces the existing mass data query techniques such as index, SQL statement optimization, data prefetching, approximate matching and distributed query, and summarizes the application range of various technologies. Then, the classical top-k query algorithm is analyzed, based on the research of TA (Threshold algorithm) algorithm and NRA (No Random Access) algorithm and approximate matching query idea, a new top-k algorithm based on extraction is proposed (top-k Algorithm Based on Extraction,tabe), the algorithm first extracts the optimal tuples and then runs the query algorithm for these tuples. In order to test the performance of Tabe algorithm, the test experiment was designed and compared with the classical NRA algorithm. The test results show that the TABE algorithm is not only of low time complexity, but also has high accuracy and can meet the requirements of conventional query. The paper also adapts to the parallel trend of mass data processing, realizes the Tabe algorithm in the Hadoop environment, and performs the performance test experiment, the results show that the query efficiency of TABE algorithm can be further improved with the help of cloud computing. This paper makes a useful research on the fast query of massive data.

Key words: Mass data top-k Hadoop Hive

Temp_12070408221992.zip