Hadoop 2.30 compiled in Ubuntu 14.04

Reprint please indicate author: Kiwenlau, and original address: http://www.cnblogs.com/kiwenlau/p/4227204.html

The compiled hadoop-2.3.0.tar.gz binary package provided by the Hadoop website is compiled on a 32-bit system and there are some errors running on the 64 system, such as:

WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

You need to compile your own Hadoop 2.30 source code at this point. My compiled Hadoop 2.30 binary package has been experimentally proven to correctly install and correctly run the word count program. Here you can download my compiled binary package:

Http://pan.baidu.com/s/1eQrgsWa

The following are the compilation steps

1. Update the package list

Apt-get Update

2. Install the software required for compiling: Why do I need these dependent packages? I don't know, either.

Apt-get install-y openjdk-7-jdk libprotobuf-dev protobuf-compiler maven cmake build-essential pkg-config libssl-dev zlib 1g-dev LLVM-GCC Automake autoconf make

3. Download the source package for Hadoop 2.30

wget http://archive.apache.org/dist/hadoop/core/hadoop-2.3.0/hadoop-2.3.0-src.tar.gz

4. Extract the source package from Hadoop 2.30

TAR-XZVF hadoop-2.3.0-src.tar.gz

5. Go to the Hadoop 2.30 folder

CD HADOOP-2.3.0-SRC

6. Compiling Hadoop 2.30 source files

MVN Package-pdist,native-dskiptests–dtar

The results of the correct execution are as follows:

[INFO]------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ......... ............... SUCCESS [1:11.968s]

[INFO] Apache Hadoop Project POM ......... ......... SUCCESS [30.393s]

[INFO] Apache Hadoop Annotations ......... .......... SUCCESS [18.398s]

[INFO] Apache Hadoop assemblies ......... .......... SUCCESS [0.246s]

[INFO] Apache Hadoop Project Dist POM ......... ..... SUCCESS [20.372s]

[INFO] Apache Hadoop Maven Plugins ........ ......... SUCCESS [23.721s]

[INFO] Apache Hadoop minikdc ......... ............. SUCCESS [1:41.836s]

[INFO] Apache Hadoop Auth ......... ............... SUCCESS [22.303s]

[INFO] Apache Hadoop Auth Examples ........ ......... SUCCESS [7.052s]

[INFO] Apache Hadoop Common ......... ............. SUCCESS [2:29.466s]

[INFO] Apache Hadoop NFS ......... ................ SUCCESS [11.604s]

[INFO] Apache Hadoop Common Project ......... ....... SUCCESS [0.073s]

[INFO] Apache Hadoop HDFS ......... ............... SUCCESS [1:30.230s]

[INFO] Apache Hadoop Httpfs ......... ............. SUCCESS [17.976s]

[INFO] Apache Hadoop HDFS bookkeeper Journal .... ..... SUCCESS [19.927s]

[INFO] Apache Hadoop Hdfs-nfs ......... ............ SUCCESS [3.304s]

[INFO] Apache Hadoop HDFS Project ......... ......... SUCCESS [0.032s]

[INFO] Hadoop-yarn ..... ....... ................... SUCCESS [0.033s]

[INFO] Hadoop-yarn-api .......... ................. SUCCESS [36.284s]

[INFO] Hadoop-yarn-common ......... ............... SUCCESS [33.912s]

[INFO] hadoop-yarn-server ......... ............... SUCCESS [0.213s]

[INFO] Hadoop-yarn-server-common ......... ......... SUCCESS [8.193s]

[INFO] Hadoop-yarn-server-nodemanager ......... ..... SUCCESS [41.181s]

[INFO] hadoop-yarn-server-web-proxy ......... ....... SUCCESS [2.768s]

[INFO] Hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [13.923s]

[INFO] hadoop-yarn-server-tests ......... .......... SUCCESS [0.904s]

[INFO] hadoop-yarn-client ......... ............... SUCCESS [4.363s]

[INFO] hadoop-yarn-applications ......... .......... SUCCESS [0.120s]

[INFO] Hadoop-yarn-applications-distributedshell ..... SUCCESS [2.262s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.615s]

[INFO] hadoop-yarn-site .......... ................ SUCCESS [0.086s]

[INFO] hadoop-yarn-project ......... .............. SUCCESS [2.703s]

[INFO] hadoop-mapreduce-client ......... ........... SUCCESS [0.132s]

[INFO] Hadoop-mapreduce-client-core ......... ....... SUCCESS [18.951s]

[INFO] Hadoop-mapreduce-client-common ......... ..... SUCCESS [14.320s]

[INFO] hadoop-mapreduce-client-shuffle ......... ..... SUCCESS [3.330s]

[INFO] Hadoop-mapreduce-client-app ........ ......... SUCCESS [9.664s]

[INFO] hadoop-mapreduce-client-hs ......... ......... SUCCESS [7.678s]

[INFO] hadoop-mapreduce-client-jobclient ....... ..... SUCCESS [9.263s]

[INFO] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [1.549s]

[INFO] Apache Hadoop MapReduce Examples ........ ..... SUCCESS [5.748s]

[INFO] hadoop-mapreduce .......... ................ SUCCESS [2.880s]

[INFO] Apache Hadoop MapReduce streaming ....... ..... SUCCESS [7.080s]

[INFO] Apache Hadoop distributed Copy ......... ..... SUCCESS [14.648s]

[INFO] Apache Hadoop Archives ......... ............ SUCCESS [2.602s]

[INFO] Apache Hadoop rumen ......... .............. SUCCESS [5.706s]

[INFO] Apache Hadoop gridmix ......... ............. SUCCESS [3.649s]

[INFO] Apache Hadoop Data Join ......... ........... SUCCESS [2.483s]

[INFO] Apache Hadoop Extras ......... ............. SUCCESS [2.678s]

[INFO] Apache Hadoop Pipes ......... .............. SUCCESS [6.359s]

[INFO] Apache Hadoop OpenStack support ....... ....... SUCCESS [5.088s]

[INFO] Apache Hadoop Client ......... ............. SUCCESS [4.534s]

[INFO] Apache Hadoop mini-cluster ......... ......... SUCCESS [0.433s]

[INFO] Apache Hadoop Scheduler Load Simulator ...... SUCCESS [7.757s]

[INFO] Apache Hadoop Tools Dist ......... .......... SUCCESS [4.099s]

[INFO] Apache Hadoop Tools ......... .............. SUCCESS [0.428s]

[INFO] Apache Hadoop distribution ......... ......... SUCCESS [18.045s]

[INFO]------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO]------------------------------------------------------------------------

[INFO] Total time:14:59.240s

[INFO] Finished At:thu Jan 18:51:59 JST 2015

[INFO] Final memory:168m/435m

[INFO]------------------------------------------------------------------------

The compiled binary package is located in the

Hadoop-2.3.0-src/hadoop-dist/target/hadoop-2.3.0.tar.gz

PS: When installing Hadoop 2.30 with a self-compiled Hadoop 2.30 binary package, you need to be aware that the following two lines are removed from the. bashrc file and the hadoop-env.sh file (these two lines are not available by default, but may be rewritten when trying to resolve an error)

Export hadoop_common_lib_native_dir= "~/hadoop/lib/"

Export hadoop_opts= "$HADOOP _opts-djava.library.path=~/hadoop/lib/"

Hadoop 2.30 is compiled in Ubuntu 14.04