Hadoop's computing capability scheduling algorithm has been being modified over the past few weeks and has encountered such a problem.
My modified version is hadoop-0.20.2
Step 1:
Load hadoop source code into eclipse and use ant to compile
Step 2:
Modify source code as needed
Step 3:
To use ant to compile and modify the content, make sure that the JDK on the compiling platform and the JDK on the running platform are the same.
The ant compilation method is: first go to the $ hadoophome/src/contrib/Capacity-scheduler directory
You can directly input ant to compile the JAR file. After compilation is successful, the generated JAR file is displayed in $ hadoophome/build/contrib/Capacity-scheduler.
It is important that if you modify other content (mainly in mapreduce) than capacity-scheduler, you also need to compile the modified entire hadoop
It also needs to be compiled before capacity-scheduler. Compile the entire hadoop and you will see the hadoop-0.20.2-dev-capacity-scheduler.jar under the build folder
Step 4:
Rename hadoop-0.20.2-dev-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar
And rename hadoop-$ {version}-capacity-scheduler.jar to hadoop-0.20.2-capacity-scheduler.jar
Overwrite the jar in the cluster and restart the cluster.
The above is a successful step, but there will always be errors throughout the process. The following is a detailed analysis of the errors I encountered:
I compiled a cluster in Ubuntu in windows. At the beginning, the two platforms had different JDK versions,
Cause: Java. Lang. unsupportedclassversionerror error, so ensure JDK consistency
Sometimes you may encounter org. Apache. hadoop. HDFS. server. datanode. datanode: Java. Io. ioexception: incompatible build versions: namenode bv =; datanode bv = 911707
The reason for this error is that the datanode version is different from the namenode version. I rename the generated hadoop-0.20.2-dev-capacity-scheduler.jar and overwrite it to $ hadoophome in datanode