Using MAVEN to compile Hadoop 2.2 on Linux

using Maven on Linux compiling for Hadoop 2.2

0, Environment Introduction:

Hadoop Information:

version:2.2

：

http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.2.0/

Source Package: hadoop-2.2.0-src.tar.gz

Package: hadoop-2.2.0.tar.gz

OS Environment:

[Email protected] ~]# lsb_release-a

LSB Version: : base-4.0-amd64:base-4.0-noarch:core-4.0

Distributor ID: CentOS

Description: CentOS Release 6.4 (Final)

Release: 6.4

Codename: Final

Unzip the source package:

TAR-ZXVF hadoop-2.2.0-src.tar.gz

Where the BUILDING.txt file is a description of the compilation, start preparing for the compilation now.

1. Prerequisites

* Unix System

* JDK 1.6+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running Findbugs)

* Protocolbuffer 2.5.0

* CMake 2.6 or newer (if compiling native code)

* Internet Connection for first build (to fetch all Maven and Hadoop dependencies) (PS: Requires an Internet connection)

2. Configuring the required software Environment

Pre-conditions mentioned in the OS requirements we have met, need to connect to the network to add good.

The other packages are as follows:

apache-maven-3.0.5-bin.tar.gz cmake-3.0.2.tar.gz findbugs-2.0.3.tar.gz hadoop-2.2.0-src.tar.gz JDK -7u51-linux-x64.tar.gz protobuf-2.5.0.tar.gz

Unzip the package above using TAR-ZXVF PackageName.

For Maven, JDK, findbugs you only need to configure the corresponding directories in the environment variables.

Vim/etc/profile Export maven_home=/u01/app/apache-maven-3.0.5 export java_home=/u01/app/jdk1.7.0_51 export F indbugs_home1=/u01/app/findbugs-2.0.3 Export Path=${findbugs_home1}/bin:${java_home}/bin:${maven_home}/bin: $PATH

CMake installation can refer to the README.txt file, the following command can be used to complete the installation:./bootstrap && make && make install

Protobuf the README.txt file in the installation reference package, use the following command to complete the installation:

$./configure $ make $ do check $ make install

Through the above steps we can complete the configuration of the pre-conditions.

3. Bug with 2.2

Reference Links: https://issues.apache.org/jira/browse/HADOOP-10110

Hadoop 2.2 has a bug that affects compilation, and we do this here in advance to deal with the method:

In the Hadoop-common-project/hadoop-auth/pom.xml file     <dependency>       <groupId>org.mortbay.jetty</groupId>      < Artifactid>jetty</artifactid>      <scope>test</scope>     </dependency> add-on behind     <dependency>       <groupId>org.mortbay.jetty</groupId>      < Artifactid>jetty-util</artifactid>      <scope>test</scope>     </dependency>    in the Hadoop-hdfs-project/hadoop-hdfs/pom.xml file     <groupId>org.apache.maven.plugins</groupId>   <artifactId> maven-javadoc-plugin</artifactid>   <configuration>       <excludepackagenames>org.apache.hadoop.hdfs.protocol.proto</excludepackagenames>   </configuration>     Behind plus    <executions><execution><id>attach-javadocs</id><goals ><goal>jar</goal></goals></execution>   </executions>

4. compiling

MAVEN's specific compilation commands we can find in BUILDING.txt, as follows:

Building Distributions:

Create binary distribution without native code and without documentation:

$ MVN Package-pdist-dskiptests-dtar

Create binary distribution with native code and with documentation:

$ MVN Package-pdist,native,docs-dskiptests-dtar

Create Source Distribution:

$ MVN package-psrc-dskiptests

Create source and binary distributions with native code and documentation:

$ MVN Package-pdist,native,docs,src-dskiptests-dtar

Create a local staging version of the website (in/tmp/hadoop-site)

$MVN clean site;

MVN Site:stage-dstagingdirectory=/tmp/hadoop-site

Here we compile with mvn Package-pdist,native,docs,src-dskiptests-dtar, which can be run repeatedly during compilation due to a network outage. Here we can enter the sub-package to compile the department (equivalent to download dependent packages separately), and then run the command in the HADOOP-2.2.0-SRC directory.

5. Compile Complete

After compiling, we enter the Hadoop-2.2.0-src/hadoop-dist directory, we can find a target directory, in which we can find the compiled compressed package.

Hadoop-2.2.0.tar.gz

This article from the "Ah Cloth" blog, declined reprint!

Using MAVEN to compile Hadoop 2.2 on Linux