Design and implementation of dynamic load balancing scheduling algorithm based on feedback in Hadoop heterogeneous environment

Design and implementation of dynamic load balancing scheduling algorithm based on feedback in Hadoop heterogeneous environment

Nanjing University of Li Yuanhong

On the basis of introducing the basic concept, architecture and application development of cloud computing, this paper analyzes the implementation framework and fault-tolerant mechanism of cloud computing model. For the cloud computing Hadoop Open source platform, the Distributed File System (HDFS, Hadoop Distributed File systems) and MapReduce computing model are analyzed in detail, and the job scheduling technology, FIFO, The job scheduling algorithm of fair queue and computational ability has been studied deeply. The computing performance of the existing scheduling algorithms in Hadoop is analyzed in detail, including the CPU utilization of hardware resources and the influence of disk I/O read and write frequency on MapReduce job scheduling. On the basis of this, a dynamic load balancing scheduling algorithm based on feedback is proposed by improving the existing algorithm of computing capacity scheduling.

Key words: Cloud computing Hadoop MapReduce Job scheduling heterogeneous cluster load balancing

[Download Address]:http://bbs.chinacloud.cn/showtopic-15560.aspx