Hierarchical multi-table connection based on two-dimensional node matrix in cloud environment

Hierarchical multi-table connection based on two-dimensional node matrix in cloud environment

Tao Yongzai Zhou Mengxue Benji Wei Lin Cao Yunjie

With the advent of the "Big Data" era, distributed data processing has been widely used and developed. In cloud computing, the demand for complex processing is increasing, and data analysis usually needs to span multiple datasets. Therefore, an efficient multiple-table connection mechanism is urgently needed. The existing multi-table connection mechanism based on MapReduce adopts serial cascade to realize multiple connections of different data sets, and the operation is flexible but inefficient. Based on the analysis of the existing parallel connection model, this paper A hierarchical multi-table join model based on two-dimension node matrix is proposed TD-HMJ.TD-HMJ in the process of a map to handle all the connection properties, the reduce process to establish a two-dimensional node matrix to achieve multiple groups of 3 (or 2) Table parallel Connection, Multi-group connection is realized through the multilevel reduce process. Theoretical analysis and experiments show that TD-HMJ reduces the data transmission, shortens the time of multiple tables joining and improves the connection efficiency.

Hierarchical multi-table connection based on two-dimensional node matrix in cloud environment