Modify hadoop Job Scheduling Algorithm Process Analysis

Hadoop's computing capability scheduling algorithm has been being modified over the past few weeks and has encountered such a problem.

My modified version is hadoop-0.20.2

Step 1:

Load hadoop source code into eclipse and use ant to compile

Step 2:

Modify source code as needed

Step 3:

To use ant to compile and modify the content, make sure that the JDK on the compiling platform and the JDK on the running platform are the same.

The ant compilation method is: first go to the $ hadoophome/src/contrib/Capacity-scheduler directory

You can directly input ant to compile the JAR file. After compilation is successful, the generated JAR file is displayed in $ hadoophome/build/contrib/Capacity-scheduler.

It is important that if you modify other content (mainly in mapreduce) than capacity-scheduler, you also need to compile the modified entire hadoop

It also needs to be compiled before capacity-scheduler. Compile the entire hadoop and you will see the hadoop-0.20.2-dev-capacity-scheduler.jar under the build folder

Step 4:

Rename hadoop-0.20.2-dev-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar

And rename hadoop-$ {version}-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar

Overwrite the jar in the cluster and restart the cluster.

The above is a successful step, but there will always be errors throughout the process. The following is a detailed analysis of the errors I encountered:

I compiled a cluster in Ubuntu in windows. At the beginning, the two platforms had different JDK versions,

Cause: Java. Lang. unsupportedclassversionerror error, so ensure JDK consistency

Sometimes you may encounter org. Apache. hadoop. HDFS. server. datanode. datanode: Java. Io. ioexception: incompatible build versions: namenode bv =; datanode bv = 911707

The reason for this error is that the datanode version is different from the namenode version. I rename the generated hadoop-0.20.2-dev-capacity-scheduler.jar and overwrite it to $ hadoophome in datanode