Skyline calculation based on the MapReduce projection partition under the frame

Skyline calculation based on the MapReduce projection partition under the frame

Wang Shuyan Yang Xin Lickerich

Recent， Skyline Computing plays a more and more important role in decision application. The research on single machine processing has been more mature. Nowadays large data explosion, Skyline Computing is confronted with the problem of data processing. The MapReduce is a parallel model, which is widely used in the application processing. As we all know, mapreduce processing requires that tasks be decomposable. Sk When the yline computation executes on the MapReduce, the method of decomposing the task has the Grid division, Based on the angle of division and so on. Grid partitioning behaves well only when the data dimension is low, and the segmentation based on angle is suitable for low and high dimensional data however, a complex and time-consuming coordinate conversion process is needed before partitioning. A data set is decomposed by a kind of segmentation based on the angle division, which is suitable for low and high dimensional data, and it is simpler to transform the coordinates before the partition. Single. Based on the partition of the hyperplane projection, an algorithm for Skyline computation on MapReduce is proposed MR-HPP (MapReduce with hyperplane-projections-based partition), An effective filtering algorithm, PSF (presorting filter), is proposed in the filtering phase of the algorithm. A large number of comparison experiments based on the Hadoop platform show that the algorithm is accurate, efficient and stable.

Skyline calculation based on the MapReduce projection partition under the frame