Research and optimization of mapreduce high availability

Research and optimization of mapreduce high availability

Huang Weijian Zhou Yi Love

In order to improve the usability of MapReduce, a distributed jobtracker Node model optimization scheme is proposed. One of the core technologies of Hadoop mapreduce the job scheduling process in the programming model, points out the performance bottleneck caused by single jobtracker, and puts forward the corresponding optimization scheme. The single Jobtracker node in the original MapReduce is optimized to be a jobtracker cluster with multiple jobtracker nodes; On the basis of this model, the communication method is optimized by using many pairs of communication methods, and the work balance is optimized by the way of maintaining the list of operation quantities. Improve mapreduce availability and job execution efficiency. By testing the fault tolerance and execution efficiency of the system, the high availability and efficiency of the optimized model are validated.

Research and optimization of mapreduce high availability