Getting Started with Hadoop (Hadoop compilation)

Description

Hadoop version: hadoop-2.5.0-cdh5.3.6

Environment: centos6.4

Must be networked

Hadoop Download URL: http://archive.cloudera.com/cdh5/cdh/5/

In fact, compiling is really manual work, according to the official instructions, step by step down to do it, but always meet the pit.

Compile steps :

1, download the source code, decompression, in this case, extracted to/opt/softwares:

Command: TAR-ZXVF hadoop-2.5.0-cdh5.3.6-src.tar.gz-c/opt/softwares

2 , into the unpacked directory, there are BUILDING.txt, view:

Command: cd/opt/softwares/hadoop-2.5.0-cdh5.3.6

Cat BUILDING.txt

A compilation is described in the documentation, such as the following:

To compile Hadoop, the software must be installed on the system.

(1) the system is Linux and is eligible

(2) JDK version 1.6 or higher:

The machine is installed jdk1.7.0_67, the environment variable is configured. Before the installation of the JDK, it is recommended that you first look at the system is not installed, if there is installed, less than 1.6 version, it is best to first uninstall before reloading.

After loading, configure the environment variables, be sure to use java-version to see if the corresponding version number.

There is a hole here, if your previous version is not uninstalled, less than 1.6 version, then the compilation will default to the previous version, and then ~ ~ ~ You know ...

(3) Install MAVEN, native installation maven3.0.5,, official website: https://archive.apache.org/dist/maven/maven-3/

After loading, configure environment variables (refer to official instructions)

(4) FindBugs This I did not install, mainly for the purpose of finding bugs. Non-essential.

(5) Protocolbuffer, Protoc buffer is a communication tool for Hadoop.

We recommend that the environment is ready for the end, and then put this, the following will give specific instructions

(6) Installation CMake:

Command: Yum install autoconf automake libtool cmake

Show complete is installed (same as, no stickers)

(7) Install Zlib devel: Many systems are already self-brought.

Command: Yum install zlib devel

(8) Installation Ncurses-devel

Command: Yum install Ncurses-devel

(9) Installation Openssl-devel

Command: Yum install Openssl-devel

(10) Finally, be sure to install GCC-related components

Command:yum install gcc*

(11) Now look back at the installation of Protobuf:

Download protobuf-2.5.0.tar.gz: Http://pan.baidu.com/s/1pJlZubT (This is not my, reference URL: http://www.tuicool.com/articles/jM7Nn2/)

Decompression TAR-ZXVF protobuf-2.5.0.tar.gz

Go to unzip directory protobuf-2.5.0:

(1) implementation./configure

(2) Perform make

(3) Perform make check

(4) Execute make install

When you are finished, enter protoc–version, and if you return the following results, the installation succeeds:

By this, the environment is almost ready.

Now start compiling the source code:

Look back at BUILDING.txt, compile the command:

Now that you've compiled it, you need to native the local library. Document, no need to use ( Pit: The first few compilations, I used this command, MVN Package-pdist,native,docs-dskiptests-dtar

has been unsuccessful, all errors on the same issue, so far do not know how to solve . Later wearing the boss said that docs do not generate, no effect. Change to MVN Package-pdist,native-dskiptests-dtar, sure enough to compile successfully.

）

Enter the hadoop-2.5.0-cdh5.3.6 directory and execute the command:

MVN Package-pdist,native-dskiptests-dtar

Wait for the compilation to succeed.

Here, the problem that arises in the process of compiling, is mainly the problem shown in the following diagram:

In general, this problem may be due to network reasons, the first time there is no successful download, the second time to continue downloading, causing conflicts. The workaround is to go to Maven

Delete the jar package from the local library and rerun the command compilation. During the compilation, due to poor network, I have three of similar problems here, are the same way to solve.

After the compilation is successful (almost the following diagram O (∩_∩) o),

Into the Hadoop-2.5.0-cdh5.3.6/hadoop-dist/target, You can see the compiled Hadoop. If you have previously installed this version of Hadoop, but not compiled, there will be no loading of the local library warning, then you can enter into the Target/hadoop-2.5.0-cdh5.3.6/lib, Copy native in your original Hadoop corresponding to the LIB, there will be no warning.

All right, that's all, just so many things, all day. Keng

Getting Started with Hadoop (Hadoop compilation)