Hadoop 2.5.2 Source Code compilation

The compilation process is very long, the mistakes are endless, need patience and patience!! 1. Preparation of the environment and software

Operating system: Centos6.4 64-bit

JDK:JDK-7U80-LINUX-X64.RPM, do not use 1.8

Maven:apache-maven-3.3.3-bin.tar.gz

protobuf:protobuf-2.5.0.tar.gz Note: Google's products, preferably in advance Baidu prepared this document

Hadoop src:hadoop-2.5.2-src.tar.gz Hadoop's official website download

ant:apache-ant-1.9.6-bin.tar.gz when compiling common, it will be used

OpenSSL devel

Ncurses-devel

cmake:2.6 or later

The compilation process ensures that the network is unblocked and needs to download other dependent packages from the extranet.

2. Environment Preparation Process 1) Install JDK

Easier to install by RPM-IVH jdk-7u80-linux-x64.rpm 2) Extract Hadoop source Package

Tar zxvf hadoop-2.5.2-src.tar.gz

View the BUILDING.txt within the folder Hadoop-2.5.2-src, which has the required environmental requirements for compilation

VI BUILDING.txt

3) Install Maven (root user installation)

Direct decompression to directory/usr

Tar zxvf apache-maven-3.3.3-bin.tar.gz-c/usr

Setting environment variables

VI +/etc/profileexport maven_home=/usr/apache-maven-3.3.3export path=.: $PATH: $MAVEN _home/bin

Use the source command to make the configuration file effective

Source/etc/profile

Verify that Maven installs successfully mvn-version, displays information such as Maven,java,linux

4) Install Protobuf

Unzip to/usr directory

Tar zxvf protobuf-2.5.0.tar.gz-c/usr

Switch to the directory/usr/protobuf-2.5.0 and then execute the./configrue,make command

./configure

Need to install GCC C + + to perform Yum installation

Yum Install Gccyum install gcc-c++  #会两次提示输入y (yes/no) yum install make

I'm doing the second command above, configure to compile the error, after viewing the exception, C + + did not install successfully, then once again executed: Yum install gcc-c++ command, OK

After the installation is complete, execute the following make command after/configure.

Makemake Install  

The command executes for a long time, the last command executes the installation, and if an error occurs, restart it.

Test whether the installation is successful after execution completes

Protoc--version

5) Install CMake, openssl-devel

CMake requires more than 2.6 of the version, in the installation will be prompted to enter the y/n, and because the installation of this component is required to network, so according to different speed, the installation progress is not the same

Yum install cmake    yum install Openssl-develyum install Ncurses-devel

6) Install Ant

Unzip the file

Tar zxvf apache-ant-1.9.6-bin.tar.gz-c/usr

Setting environment variables

VI +/etc/profileexport ant_home=/usr/apache-ant-1.9.6export path=.: $PATH: $ANT _home/bin

Use the source command to make the configuration file effective

Source/etc/profile

Verify that the installation is successful ant-version, display version information

3. Compiling Hadoop source code

Switch to a Hadoop user

Switch to directory hadoop-2.5.2-src, execute command

MVN package-deskiptests-pdist,native

After a long wait process, the intermediate process will download a lot of files, the specific role is unknown, often because the download network problems failed to execute the compilation command again, previously downloaded will not re-execute the download

Failure occurs, if the log that was found before output [INFO] is stuck on a download link, retry directly

A compile test error has occurred, for example, if the cause is not currently identified, retry directly

After the unexpected interrupt compiles, after starting again, compile the exception, such as

To solve the problem, try again with the following command, in two steps

The first step, the implementation

command, waiting to complete (will automatically network download a lot of things) and quickly after the internet, not smooth when repeated several times

[INFO]------------------------------------------------------------------------[INFO] Reactor Summary:[info] [info ] Apache Hadoop Main ....... .................. SUCCESS [4.721 s][info] Apache Hadoop Project POM ......... ......... SUCCESS [1.135 s][info] Apache Hadoop Annotations ......... ......... SUCCESS [4.990 s][info] Apache Hadoop Project Dist POM ............ SUCCESS [0.814 s][info] Apache Hadoop assemblies ................. SUCCESS [0.552 s][info] Apache Hadoop Maven Plugins .............. SUCCESS [4.834 s][info] Apache Hadoop minikdc ......... ............. SUCCESS [4.277 s][info] Apache Hadoop Auth ..... ..... ............... SUCCESS [5.709 s][info] Apache Hadoop Auth Examples .............. SUCCESS [2.516 s][info] Apache Hadoop Common ......... .............. SUCCESS [53.258 s][info] Apache Hadoop NFS ...... ..... ............... SUCCESS [1.175 s][info] Apache Hadoop COmmon Project ......... ........ SUCCESS [0.070 s][info] Apache Hadoop HDFS ..... ..... ............... SUCCESS [01:05 min][info] Apache Hadoop httpfs ......... .............. SUCCESS [4.430 s][info] Apache Hadoop HDFS bookkeeper Journal ...... SUCCESS [28.926 s][info] Apache Hadoop hdfs-nfs ......... ............ SUCCESS [0.851 s][info] Apache Hadoop HDFS Project ......... ........ SUCCESS [0.092 s][info] hadoop-yarn ..... ........ ................... SUCCESS [0.063 s][info] hadoop-yarn-api ..... ..... .................. SUCCESS [4.983 s][info] Hadoop-yarn-common ......... ................ SUCCESS [01:01 min][info] hadoop-yarn-server ......... ................ SUCCESS [0.092 s][info] Hadoop-yarn-server-common ......... .......... SUCCESS [20.415 s][info] hadoop-yarn-server-nodemanager ............. SUCCESS [01:24 min][info] hadoop-yarn-server-web-proxy .............. SUCCESS [0.6S][info] hadoop-yarn-server-applicationhistoryservice .... SUCCESS [1.173 s][info] hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [5.231 s][info] hadoop-yarn-server-tests ......... .......... SUCCESS [1.075 s][info] hadoop-yarn-client ......... ................ SUCCESS [1.853 s][info] hadoop-yarn-applications ......... .......... SUCCESS [0.129 s][info] Hadoop-yarn-applications-distributedshell .... SUCCESS [0.563 s][info] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [0.515 s][info] hadoop-yarn-site ..... ..... ................. SUCCESS [0.112 s][info] hadoop-yarn-project ......... ............... SUCCESS [0.132 s][info] hadoop-mapreduce-client ......... ........... SUCCESS [0.151 s][info] hadoop-mapreduce-client-core .............. SUCCESS [8.638 s][info] Hadoop-mapreduce-client-common ............. SUCCESS [5.152 s][info] hadoop-mapreduce-client-shuffle ............... SUCCESS [1.165 s][info] hadoop-mapreduce-client-app ............... SUCCESS [4.623 s][info] hadoop-mapreduce-client-hs ......... ......... SUCCESS [1.714 s][info] hadoop-mapreduce-client-jobclient ....... ..... SUCCESS [18.468 s][info] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [0.856 s][info] Apache Hadoop MapReduce Examples ........ ..... SUCCESS [1.955 s][info] hadoop-mapreduce ..... ..... ................. SUCCESS [0.096 s][info] Apache Hadoop MapReduce streaming ....... ..... SUCCESS [9.471 S][info] Apache Hadoop distributed Copy ............ SUCCESS [01:08 min][info] Apache Hadoop Archives ......... ............ SUCCESS [0.841 s][info] Apache Hadoop rumen ........ ............... SUCCESS [1.113 s][info] Apache Hadoop gridmix ......... ............. SUCCESS [2.258 s][info] Apache Hadoop Data Join ......... ........... SUCCESS [0.981 S][info] Apache Hadoop Extras ......... ............... SUCCESS [1.185 s][info] Apache Hadoop Pipes ........ ............... SUCCESS [0.098 s][info] Apache Hadoop OpenStack support ........ ..... SUCCESS [1.246 s][info] Apache Hadoop Client ......... .............. SUCCESS [0.427 s][info] Apache Hadoop mini-cluster ............... SUCCESS [0.336 s][info] Apache Hadoop Scheduler Load Simulator ...... SUCCESS [21.003 s][info] Apache Hadoop Tools Dist ......... .......... SUCCESS [2.185 s][info] Apache Hadoop Tools ..... ..... ............... SUCCESS [0.097 S][info] Apache Hadoop distribution ............... SUCCESS [0.227 s][info]------------------------------------------------------------------------[INFO] BUILD Success[info]------------------------------------------------------------------------[INFO] Total time:08:47 min[ INFO] finished at:2016-01-14t09:05:59+08:00[info] Final Memory:96m/237m[info]------------------------------------------------------------------------ 

Perform the compilation

MVN Package-pdist,native-dskiptests-dtar

Compilation succeeded

[INFO] Reactor Summary:[info] [INFO] Apache Hadoop Main ....... ..... ............. SUCCESS [6.143 s][info] Apache Hadoop Project POM ......... ......... SUCCESS [3.923 s][info] Apache Hadoop Annotations ......... ......... SUCCESS [6.345 s][info] Apache Hadoop assemblies ................. SUCCESS [0.466 s][info] Apache Hadoop Project Dist POM ............ SUCCESS [3.457 s][info] Apache Hadoop Maven Plugins .............. SUCCESS [8.004 s][info] Apache Hadoop minikdc ......... ............. SUCCESS [6.740 s][info] Apache Hadoop Auth ..... ..... ............... SUCCESS [7.214 s][info] Apache Hadoop Auth Examples .............. SUCCESS [6.145 s][info] Apache Hadoop Common ......... .............. SUCCESS [02:51 min][info] Apache Hadoop NFS ...... ..... ............... SUCCESS [26.889 s][info] Apache Hadoop Common Project ......... ....... SUCCESS [0.107 s][info] Apache Hadoop HDfs................................. SUCCESS [04:44 min][info] Apache Hadoop httpfs ......... .............. SUCCESS [04:57 min][info] Apache Hadoop HDFS bookkeeper Journal ...... SUCCESS [01:01 min][info] Apache Hadoop hdfs-nfs ......... ............ SUCCESS [15.992 s][info] Apache Hadoop HDFS Project ......... ........ SUCCESS [0.307 s][info] hadoop-yarn ..... ........ ................... SUCCESS [0.160 s][info] hadoop-yarn-api ..... ..... .................. SUCCESS [02:30 min][info] Hadoop-yarn-common ......... ................ SUCCESS [01:06 min][info] hadoop-yarn-server ......... ................ SUCCESS [0.111 s][info] Hadoop-yarn-server-common ......... .......... SUCCESS [27.082 s][info] hadoop-yarn-server-nodemanager ............. SUCCESS [40.111 s][info] hadoop-yarn-server-web-proxy .............. SUCCESS [9.432 s][info] hadoop-yarn-server-applicationhistoryservice .... SUCCESS [15.2[S][info] Hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [38.984 s][info] hadoop-yarn-server-tests ......... .......... SUCCESS [0.889 s][info] hadoop-yarn-client ......... ................ SUCCESS [13.748 s][info] hadoop-yarn-applications ......... .......... SUCCESS [0.165 s][info] Hadoop-yarn-applications-distributedshell .... SUCCESS [7.001 s][info] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [4.640 s][info] hadoop-yarn-site ..... ..... ................. SUCCESS [0.177 s][info] hadoop-yarn-project ......... ............... SUCCESS [13.099 s][info] hadoop-mapreduce-client ......... ........... SUCCESS [0.141 s][info] hadoop-mapreduce-client-core .............. SUCCESS [39.011 s][info] Hadoop-mapreduce-client-common ............. SUCCESS [33.940 s][info] hadoop-mapreduce-client-shuffle ......... ..... SUCCESS [7.769 s][info] hadoop-mapreduce-client-app .................. SUCCESS [18.003 s][info] hadoop-mapreduce-client-hs ......... ......... SUCCESS [16.896 s][info] hadoop-mapreduce-client-jobclient ....... ..... SUCCESS [7.284 s][info] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [3.377 s][info] Apache Hadoop MapReduce Examples ........ ..... SUCCESS [11.164 s][info] hadoop-mapreduce ..... ..... ................. SUCCESS [5.954 s][info] Apache Hadoop MapReduce streaming ....... ..... SUCCESS [8.246 S][info] Apache Hadoop distributed Copy ............ SUCCESS [14.047 s][info] Apache Hadoop Archives ......... ............ SUCCESS [3.657 s][info] Apache Hadoop rumen ........ ............... SUCCESS [12.067 s][info] Apache Hadoop gridmix ......... ............. SUCCESS [8.998 s][info] Apache Hadoop Data Join ......... ........... SUCCESS [5.860 s][info] Apache Hadoop Extras ......... .............. SUCCESS [5.582 S][info] Apache HadoopPipes ......... ............... SUCCESS [11.003 s][info] Apache Hadoop OpenStack support ........ ..... SUCCESS [11.071 s][info] Apache Hadoop Client ......... .............. SUCCESS [16.352 s][info] Apache Hadoop mini-cluster ............... SUCCESS [0.241 s][info] Apache Hadoop Scheduler Load Simulator ...... SUCCESS [8.179 s][info] Apache Hadoop Tools Dist ......... .......... SUCCESS [7.388 s][info] Apache Hadoop Tools ..... ..... ............... SUCCESS [0.048 S][info] Apache Hadoop distribution ............... SUCCESS [01:07 min][info]------------------------------------------------------------------------[INFO] BUILD Success[info]------------------------------------------------------------------------[INFO] Total time:27:05 min[ INFO] finished at:2016-01-14t09:56:35+08:00[info] Final Memory:77m/237m[info]------------------------------------- -----------------------------------

Check the compiled local file, complete the compilation

Hadoop 2.5.2 Source Code compilation