Compile hadoop-append for hbase

HbaseBased on hadoop, if hbase uses the release version of hadoop directly, data may be lost. hbase needs to use hadoop-append. For more information, seeHbase
Official website materials

The following uses hbase-0.90.2 as an example to introduce the compilation of hadoop-0.20.2-append, the following Operation Reference:

Building an hadoop 0.20.x version for hbase 0.90.2

Operating System: centos Linux

Required tools: git, ant

[Zhankunlin @ icthtc ~] $ Mkdir hadoop-append-for-hbase
[Zhankunlin @ icthtc ~] $ CD hadoop-append-for-hbase
[Zhankunlin @ icthtc hadoop-append-for-hbase] $Git clone http://git.apache.org/hadoop-common.git
[Zhankunlin @ icthtc hadoop-append-for-hbase] $CD hadoop-common
[Zhankunlin @ icthtc hadoop-common] $Git checkout-T origin/branch-0.20-append
[Zhankunlin @ icthtc hadoop-common] $Git show-branch-0.20-append
[Zhankunlin @ icthtc hadoop-common] $Git show-branch release-0.20.2
[Zhankunlin @ icthtc hadoop-common] $Vi ../build. Properties
# This is essential
Resolvers = internal
# You can increment this number as you see fit
Version = 0.20-append-for-hbase-zkl
Project. Version =$ {version}
Hadoop. Version =$ {version}
Hadoop-core.version =$ {version}
Hadoop-hdfs.version =$ {version}
Hadoop-mapred.version =$ {version}
[Zhankunlin @ icthtc hadoop-common] $Ln-S ../build. properties build. Properties
[Zhankunlin @ icthtc hadoop-common] $ pwd
/Home/zhankunlin/hadoop-append-for-hbase/hadoop-common

Install ant
[Zhankunlin @ icthtc ~] $ Tar zxf apache-ant-1.8.2-bin.tar.gz

[Zhankunlin @ icthtc ~] $ Export Path =/home/zhankunlin/Apache-ant-1.8.2/bin: $ path

Compile hadoop

[Zhankunlin @ icthtc ~] $ CD hadoop-append-for-hbase/hadoop-common
[Zhankunlin @ icthtc hadoop-common] $Git checkout branch-0.20-appendMake sure that branch-0.20-append is currently used
Already on 'Branch-0.20-append'
[Zhankunlin @ icthtc hadoop-common] $Ant MVN-install
Buildfile:/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build. xml
.
.
.
MVN-install:
[Artifact: Install] [info] installing/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build/hadoop-core-0.20-append-for-hbase-zkl.jar to/home/zhankunlin /. m2/Repository/org/Apache/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
[Artifact: Install] [info] installing/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build/hadoop-test-0.20-append-for-hbase-zkl.jar to/home/zhankunlin /. m2/Repository/org/Apache/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
[Artifact: Install] [info] installing/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build/hadoop-tools-0.20-append-for-hbase-zkl.jar to/home/zhankunlin /. m2/Repository/org/Apache/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
[Artifact: Install] [info] installing/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build/hadoop-examples-0.20-append-for-hbase-zkl.jar to/home/zhankunlin /. m2/Repository/org/Apache/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
[Artifact: Install] [info] installing/home/zhankunlin/hadoop-append-for-hbase/hadoop-Common/build/contrib/streaming/hadoop-streaming-0.20-append-for-hbase-zkl.jar to/home/zhankunlin /. m2/Repository/org/Apache/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar

Build successful
Total time: 11 minutes 42 seconds

Optional operations:
[Zhankunlin @ icthtc hadoop-common] $Ant TestIt takes a long time to test all functions
[Zhankunlin @ icthtc hadoop-common] $Ant test-coreTest core functions

View the compiled jar package (in the current user's main directory)
[Zhankunlin @ icthtc hadoop-common] $ find ~ /. M2/repository-name "hadoop-*. Jar"
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
/Home/zhankunlin/. m2/Repository/org/Apache/hadoop-core/0.20.2/hadoop-core-0.20.2.jar

Compiled packages include:
[Zhankunlin @ icthtc hadoop-common] $ find ~ /. M2/repository-name "hadoop-*. Jar" | awk-F "/" '{print $11 }'
Hadoop-tools-0.20-append-for-hbase-zkl.jar
Hadoop-test-0.20-append-for-hbase-zkl.jar
Hadoop-examples-0.20-append-for-hbase-zkl.jar
Hadoop-streaming-0.20-append-for-hbase-zkl.jar
Hadoop-core-0.20-append-for-hbase-zkl.jar
Hadoop-core-0.20.2.jar (this pack is not needed)

Note:
Here, 0.20-append-for-hbase-zkl is the version number defined in the build. properties file, that is:

Hadoop-tools-VERSION.jar
Hadoop-test-VERSION.jar
Hadoop-examples-VERSION.jar
Hadoop-streaming-VERSION.jar
Hadoop-core-VERSION.jar

Copy the compiled jar package.
[Zhankunlin @ icthtc hadoop-common] $ MK ../OK _jars
[Zhankunlin @ icthtc hadoop-common] $ jars = 'Find ~ /. M2/repository-name "hadoop-*. Jar" '; for jar in $ jars; do CP $ jar ../OK _jars/; done

NextReplace the new jar generated by compilation with the old jar package in the hadoop-0.20.2-release and hbase-0.90.2

(1) Replace the old hadoop package;

Note that you need to rename the new jar package generated by compilation before replacing the old jar package in hadoop.

The naming rules for jar packages in hadoop 0.20.2 release are hadoop-VERSION-PACKAGE.jar, for example, hadoop-0.20.2-examples.jar.

While the newly compiled jar package naming rules for hadoop-PACKAGE-VERSION.jar, such as: hadoop-examples-0.20-append-for-hbase.jar.

Therefore, you need to rename the newly compiled jar package as follows to comply with hadoop:

Hadoop-examples-0.20-append-for-hbase-zkl.jar> hadoop-0.20-append-for-hbase-zkl-examples.jar
Hadoop-test-0.20-append-for-hbase-zkl.jar> hadoop-0.20-append-for-hbase-zkl-test.jar
Hadoop-tools-0.20-append-for-hbase-zkl.jar> hadoop-0.20-append-for-hbase-zkl-tools.jar
Hadoop-streaming-0.20-append-for-hbase-zkl.jar> hadoop-0.20-append-for-hbase-zkl-streaming.jar
Hadoop-core-0.20-append-for-hbase-zkl.jar> hadoop-0.20-append-for-hbase-zkl-core.jar

(2) Replace the package in the Lib folder of hbase

Hbase uses the naming rules for the hadoop-PACKAGE-VERSION.jar, and the new compiled jar package naming rules are consistent, so you just need to copy the jar package directly to the $ hbase_home/lib directory, do not need to rename.