Research on the optimization method of Hadoop performance based on scheduler

Research on the optimization method of Hadoop performance based on scheduler

Liu, He Chen, Tang Hong

In order to improve the scheduling performance of Hadoop Scheduler and shorten the task overall response time of Hadoop cluster, a dynamic scheduling improvement algorithm based on CPU occupancy rate is proposed. Firstly, this paper compares the traditional performance optimization methods of Hadoop, and points out that the key problem is the lack of dynamic and flexibility. On this basis, this paper analyzes the default task scheduling model of Hadoop, and proposes a new algorithm, which takes CPU occupancy rate as the load index to judge the load of the nodes according to the feedback load index, and dynamically adapts to the load change. Experimental results show that the algorithm can effectively improve the cluster performance in Hadoop cluster.

Research on the optimization method of Hadoop performance based on scheduler