Modify hadoop Job Scheduling Algorithm Process Analysis

Source: Internet
Author: User

Hadoop's computing capability scheduling algorithm has been being modified over the past few weeks and has encountered such a problem.

My modified version is hadoop-0.20.2

Step 1:

Load hadoop source code into eclipse and use ant to compile

Step 2:

Modify source code as needed

Step 3:

To use ant to compile and modify the content, make sure that the JDK on the compiling platform and the JDK on the running platform are the same.

The ant compilation method is: first go to the $ hadoophome/src/contrib/Capacity-scheduler directory

You can directly input ant to compile the JAR file. After compilation is successful, the generated JAR file is displayed in $ hadoophome/build/contrib/Capacity-scheduler.

It is important that if you modify other content (mainly in mapreduce) than capacity-scheduler, you also need to compile the modified entire hadoop

It also needs to be compiled before capacity-scheduler. Compile the entire hadoop and you will see the hadoop-0.20.2-dev-capacity-scheduler.jar under the build folder

Step 4:

Rename hadoop-0.20.2-dev-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar

And rename hadoop-$ {version}-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar

Overwrite the jar in the cluster and restart the cluster.

The above is a successful step, but there will always be errors throughout the process. The following is a detailed analysis of the errors I encountered:

I compiled a cluster in Ubuntu in windows. At the beginning, the two platforms had different JDK versions,

Cause: Java. Lang. unsupportedclassversionerror error, so ensure JDK consistency

Sometimes you may encounter org. Apache. hadoop. HDFS. server. datanode. datanode: Java. Io. ioexception: incompatible build versions: namenode bv =; datanode bv = 911707

The reason for this error is that the datanode version is different from the namenode version. I rename the generated hadoop-0.20.2-dev-capacity-scheduler.jar and overwrite it to $ hadoophome in datanode

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.