Research on job scheduling algorithm in Hadoop platform

Source: Internet
Author: User
Keywords Hadoop
Tags backup content finds hadoop platform research scheduling scheduling algorithm

Research on job scheduling algorithm in Hadoop platform

Zhaoxiaobing of Zhengzhou University

The main content of this paper is the improvement of job scheduling algorithm on Hadoop. An improved algorithm (Btis) for task backup scheduling is proposed for late algorithm to estimate task schedule value and SAMR algorithm does not consider the problem of backup execution node. The Btis algorithm accurately calculates the progress of the task through the history record, find real slow tasks that need to start a backup; When you select a fast node to start a backup for a slow task, you will consider the success rate of the work node's execution and the current load on the work node, and the node with the high success load will be eligible to perform the backup. In the autonomous Hadoop cluster, it is verified that the Btis algorithm can perform the scheduling of the user's job, and shorten the complete time of the whole operation. Some of the data in the experiment are obtained by means of averaging. By comparing the Btis algorithm with the late algorithm and the SAMR algorithm, it can be seen that the Btis algorithm is able to better determine the proportion of each phase of the task, find the most suitable to start the backup of the slow task, and the high efficiency of the backup, can shorten the completion time of the whole operation, improve the utilization Optimize platform performance.


Research on job scheduling algorithm in Hadoop platform

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.