Study on Load Balancing optimization in MapReduce

Study on Load Balancing optimization in MapReduce

Hong Min Lau Zhao Liu Yuanyuan Hong

Data analysis and processing is an important task in large-scale distributed data processing applications. Because of its simplicity and flexibility, the MapReduce programming model is becoming the core model of large-scale distributed data processing systems such as Hadoop systems. Because the data being processed may not be evenly divided, the MapReduce programming model may have data skew problems when it handles connection operations. The data skew problem severely reduces the efficiency of mapreduce to perform connection operations. Aiming at the problem of data skew in the connection operation in MapReduce, this paper analyzes the causes of the bottleneck of the MapReduce connection performance and sets up the load balance cost model, and puts forward a strategy of controlling the data skew in the connection process by using the range segmentation method to realize the load balance. The experimental results show that the proposed method obviously improves the efficiency of the connection.

Study on Load Balancing optimization in MapReduce