How to compile Apache Hadoop2.6.0 source code

how to compile Apache Hadoop2.6.0 source code

1. Installing CentOS

I am using CentOS6.5, is http://mirror.neu.edu.cn/centos/6.5/isos/x86_64/, choose Centos-6.5-x86_64-bin-dvd1.iso Download, note is 64 bit, The size is 4GB and needs to be downloaded for some time. In fact, the 6.x version can be, not necessarily 6.5.

I'm using a VMware virtual machine that allocates 2GB of memory and 20GB of disk space. The memory is too small to be slow, the disk is too small, and there may be a lack of space at compile time. The above is not the minimum configuration, according to your own machine configuration modification it. Also, be sure to keep the Linux networking state.

Following is a variety of software, I downloaded the software after all copied to the/usr/local directory, the following command to execute the path is in the/usr/local directory. Please note that the reader must pay attention to the path when reading. 2. installation JDK

Hadoop is written in Java, and the JDK must be installed to compile Hadoop.

Download the JDK from Oracle's website, http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html, select jdk-7u45-linux-x64.tar.gz download.

Execute the following command to unzip the JDK

TAR-ZXVF jdk-7u45-linux-x64.tar.gz

A folder jdk1.7.0_45 is generated and then set in the environment variable.

Execute command vi/etc/profile, add the following to the configuration file, the results are shown as follows

3. installation maven

The Hadoop source code is managed using MAVEN organization and must be downloaded by maven. Download from MAVEN official website, is http://maven.apache.org/download.cgi, choose apache-maven-3.3.3-bin.tar.gz Download.

Execute the following command to unzip the JDK

TAR-ZXVF apache-maven-3.3.3-bin.tar.gz

A folder apache-maven-3.3.3 is generated and then set in the environment variable.

Execute the command vi/etc/profile, and edit the result as shown

4. installation Protoc

Hadoop uses protocol buffer communication, download PROTOC from PROTOC official website, is https://code.google.com/p/protobuf/downloads/list, Select protobuf-2.5.0.tar.gz Download.

In order to compile and install PROTOC, several tools need to be downloaded to execute the following commands sequentially

Yum-y Install make

If the operating system is CentOS6.5 then GCC and make are already installed. Other versions are not necessarily.

Then execute the following command to extract the Protobuf

TAR-ZXVF protobuf-2.5.0.tar.gz

Generates a folder protobuf-2.5.0, and executes the following command to compile the protobuf.

Make && make install

As long as it doesn't go wrong.

After execution, the compiled file is located in the/usr/local/protoc/directory, we set the environment variable

Execute the command vi/etc/profile, and edit the result as shown

5. Install other dependencies

Sequentially execute the following command

Yum-y Install Ncurses-devel

Installation is complete. 6. compiling hadoop2.6 Source Code

Download the 2.6 stable version from the Hadoop website, which is http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.6.0/hadoop-2.6.0-src.tar.gz.

Execute the following command to unzip the JDK

TAR-ZXVF hadoop-2.6.0-src.tar.gz

A folder Hadoop-2.6.0-src is generated.

Okay, now go into the directory/usr/local/hadoop-2.6.0-src, execute the command

Cd/usr/local/hadoop-2.6.0-src

MVN package-dskiptests-pdist,native

This command will download the dependent jar from the extranet and compile the Hadoop source code, which takes a long time and you can eat.

After waiting for n long, you can see the following results:

[INFO]------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ......... ................ SUCCESS [4.414 S]

[INFO] Apache Hadoop Project POM ......... .......... SUCCESS [3.132 S]

[INFO] Apache Hadoop Annotations ......... .......... SUCCESS [5.377 S]

[INFO] Apache Hadoop assemblies ......... ........... SUCCESS [0.623 S]

[INFO] Apache Hadoop Project Dist POM ........ ....... SUCCESS [3.624 S]

[INFO] Apache Hadoop Maven Plugins ......... ......... SUCCESS [7.253 S]

[INFO] Apache Hadoop minikdc ......... ............. SUCCESS [5.040 S]

[INFO] Apache Hadoop Auth ......... ................ SUCCESS [9.449 S]

[INFO] Apache Hadoop Auth Examples ......... ......... SUCCESS [5.894 S]

[INFO] Apache Hadoop Common ......... .............. SUCCESS [02:35 min]

[INFO] Apache Hadoop NFS ......... ................. SUCCESS [9.395 S]

[INFO] Apache Hadoop KMS ..... ..... .................. SUCCESS [12.661 S]

[INFO] Apache Hadoop Common Project ......... ........ SUCCESS [0.064 S]

[INFO] Apache Hadoop HDFS ......... ................ SUCCESS [02:58 min]

[INFO] Apache Hadoop Httpfs ......... .............. SUCCESS [20.099 S]

[INFO] Apache Hadoop HDFS bookkeeper Journal ..... ..... SUCCESS [8.216 S]

[INFO] Apache Hadoop Hdfs-nfs ......... ............. SUCCESS [5.086 S]

[INFO] Apache Hadoop HDFS Project ......... ......... SUCCESS [0.061 S]

[INFO] Hadoop-yarn ..... ..... ...................... SUCCESS [0.091 S]

[INFO] Hadoop-yarn-api ........ ................... SUCCESS [01:45 min]

[INFO] Hadoop-yarn-common .......... ............... SUCCESS [38.766 S]

[INFO] hadoop-yarn-server .......... ............... SUCCESS [0.131 S]

[INFO] Hadoop-yarn-server-common ......... .......... SUCCESS [14.831 S]

[INFO] Hadoop-yarn-server-nodemanager ........ ....... SUCCESS [25.612 S]

[INFO] hadoop-yarn-server-web-proxy ........ ......... SUCCESS [6.043 S]

[INFO] hadoop-yarn-server-applicationhistoryservice .... SUCCESS [8.443 S]

[INFO] Hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [29.911 S]

[INFO] hadoop-yarn-server-tests ......... ........... SUCCESS [8.606 S]

[INFO] hadoop-yarn-client .......... ............... SUCCESS [10.038 S]

[INFO] hadoop-yarn-applications ......... ........... SUCCESS [0.118 S]

[INFO] Hadoop-yarn-applications-distributedshell ..... SUCCESS [3.389 S]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [2.003 S]

[INFO] hadoop-yarn-site .......... ................. SUCCESS [0.056 S]

[INFO] hadoop-yarn-registry ......... .............. SUCCESS [6.715 S]

[INFO] hadoop-yarn-project ......... ............... SUCCESS [3.798 S]

[INFO] hadoop-mapreduce-client ......... ............ SUCCESS [0.218 S]

[INFO] Hadoop-mapreduce-client-core ........ ......... SUCCESS [40.412 S]

[INFO] Hadoop-mapreduce-client-common ........ ....... SUCCESS [24.370 S]

[INFO] hadoop-mapreduce-client-shuffle ......... ..... SUCCESS [10.642 S]

[INFO] Hadoop-mapreduce-client-app ......... ......... SUCCESS [12.325 S]

[INFO] hadoop-mapreduce-client-hs ......... ......... SUCCESS [13.119 S]

[INFO] hadoop-mapreduce-client-jobclient ........ ..... SUCCESS [6.762 S]

[INFO] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [1.958 S]

[INFO] Apache Hadoop MapReduce Examples ....... ....... SUCCESS [8.129 S]

[INFO] hadoop-mapreduce .......... ................. SUCCESS [3.937 S]

[INFO] Apache Hadoop MapReduce streaming ........ ..... SUCCESS [5.881 S]

[INFO] Apache Hadoop distributed Copy ........ ....... SUCCESS [10.755 S]

[INFO] Apache Hadoop Archives ......... ............. SUCCESS [2.511 S]

[INFO] Apache Hadoop rumen ......... ............... SUCCESS [8.135 S]

[INFO] Apache Hadoop gridmix ......... ............. SUCCESS [5.524 S]

[INFO] Apache Hadoop Data Join ......... ............ SUCCESS [3.702 S]

[INFO] Apache Hadoop Ant Tasks ......... ............ SUCCESS [2.582 S]

[INFO] Apache Hadoop Extras ......... .............. SUCCESS [3.400 S]

[INFO] Apache Hadoop Pipes ......... ............... SUCCESS [7.537 S]

[INFO] Apache Hadoop OpenStack support ......... ..... SUCCESS [7.347 S]

[INFO] Apache Hadoop Amazon Web Services support .... SUCCESS [8.864 S]

[INFO] Apache Hadoop Client ......... .............. SUCCESS [5.480 S]

[INFO] Apache Hadoop mini-cluster ......... .......... SUCCESS [0.084 S]

[INFO] Apache Hadoop Scheduler Load Simulator ....... SUCCESS [5.272 S]

[INFO] Apache Hadoop Tools Dist ......... ........... SUCCESS [6.860 S]

[INFO] Apache Hadoop Tools ......... ............... SUCCESS [0.026 S]

[INFO] Apache Hadoop distribution ......... .......... SUCCESS [31.834 S]

[INFO]------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO]------------------------------------------------------------------------

[INFO] Total time:15:44 min

[INFO] Finished at:2015-07-13t00:23:42-07:00

[INFO] Final memory:101m/326m

[INFO]------------------------------------------------------------------------

[Email protected] hadoop-2.6.0-src]#

Well, the compilation is done.

The compiled code is below/usr/local/hadoop-2.6.0-src/hadoop-dist/target, as in.

7. precautions for hands-on

This is all the information I have compiled, the various packages used in the compilation process, here are all. The structure is as follows

If you are unfamiliar with Linux, be sure to use our recommended centos6.5 64-bit operating system. Because the various operations described in this article are for this version of the operating system.

The jar dependencies required during the compilation, I have all downloaded, and packaged, can be replaced with my content. The default location of the Maven repository in ~/.m2/repository, everyone unzip my repository to replace their own on the line.

Important NOTE: be sure to ensure that the virtual machine network unblocked . a) pseudo-distribution configuration

1. hadoop-dist-2.6.0-binary-64.tar.gz hadoop-2.6.0 extracted from the "64-bit version of the compiled hadoop2.6.0" from the folder is placed in the/usr/local directory. When placed, the complete directory structure is/usr/local/hadoop-2.6.0

If the source code is compiled, the path here refers to the

2. The configuration content in the folder "hadoop2.6.0 pseudo distribution Profile" is a pseudo-distribution setting. Copy all the contents of this directory to the/usr/local/hadoop-2.6.0/etc/hadoop directory, overwriting the original file.

3. Modify the value of the hdfs://crxy213.crxy:9000 in the Core-site.xml and change it to its own IP or host name

4. Format, execute command/USR/LOCAL/HADOOP-2.6.0/SBIN/HDFS Namenode-format

5. Start, execute script/usr/local/hadoop-2.6.0/sbin/start-hadoop.sh

----------------------------Bug

ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.0:descriptor (Default-descriptor) on project Hadoop-maven-plugins:the API of the Mojo scanner is not compatible with this plugin version. Check the plugin dependencies configured in the POM and ensure the versions match. /usr/java/jdk1.7.0_79/jre/lib/i386/xawt/libmawt.so:libxext.so.6:cannot open Shared object file:no such file or Directo Ry, [Help 1]
[ERROR]
[ERROR] to see the full stack trace of the errors, re-run Maven with the-e switch.
[ERROR] Re-run Maven using The-x switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:< Br>[error] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

Yum List | grep libxext
Yum install libxext.i686.
After I tried, no, my friend came over and said, is there a problem with the environment variable?
I checked the next JDK to be 32-bit.
After the installation of 64, and so on for 1 hours, success!!!

How to compile Apache Hadoop2.6.0 source code