Linux source code Compilation of Hadoop

Hadoop begins, according to the Convention, first compile the source code, import into Eclipse, so that later to understand that piece, or that block of problems, directly find the source code.

hadoop2.4.1 compilation requires protoc2.5.0 support, so download the PROTOC as well. I downloaded:protobuf-2.5.0.tar.bz2

Install a few dependent packages before compiling the PROTOC for the installation: Gcc,gcc-c++,make if it is already installed can be ignored

Yum Install Gccyum install Gcc-c++yum install Makeyum install cmake  yum install openssl-devel  yum install ncurses- Devel  

Installing PROTOC

TAR-XVF protobuf-2.5.0.tar.bz2  cd protobuf-2.5.0  ./configure--prefix=/opt/protoc/make  && make Install

The Linux system executes the compile command:mvn install Eclipse:eclipse-pdist,native-dskiptests-dtar -dmaven.javadoc.skip=true

When the compilation is complete, view the Hadoop-dist folder:

[Email protected] target]# lltotal 153824drwxr-xr-x. 2 root root      4096 Jul  9 17:00 antrun-rw-r--r--. 1 root root      4809 Jul  9 17:00 dist-layout-stitching.sh-rw-r --r--. 1 root root       666 Jul  9 17:01 dist-tar-stitching.shdrwxr-xr-x. 9 root root      4096 Jul  9 17:00 hadoop-3.0.0-sn Apshot-rw-r--r--. 1 root root 157482988 Jul  9 17:01 hadoop-3.0.0-snapshot.tar.gz-rw-r--r--. 1 root root      3445 Jul  9 17:01 Hadoop -dist-3.0.0-snapshot.jardrwxr-xr-x. 2 root root      4096 Jul  9 17:01 maven-archiverdrwxr-xr-x. 2 root root      4096 Jul  9 17:00 test-dir[[email PR Otected] target]# Pwd/home/fish/hadoop/hadoop-dist/target

to view the version of Hadoop:

[Email protected] bin]# Cd/home/fish/hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin[[email protected] bin]#. Hadoop versionhadoop 3.0.0-snapshotsource Code repository Https://github.com/apache/hadoop.git-r E0febce0e74ec69597376774f771da46834c42b1compiled by Root on 2015-07-09t08:53zcompiled with Protoc 2.5.0From source with Checksum d69dd13fde158d22d95a263a0f12bc8this command was run using/home/fish/hadoop/hadoop-dist/target/ Hadoop-3.0.0-snapshot/share/hadoop/common/hadoop-common-3.0.0-snapshot.jar[[email protected] bin]# pwd/home/fish/ Hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin

To view some information about the compilation:

[[email protected] hadoop-3.0.0-snapshot]# file lib//native/*lib//native/libhadoop.a:current ar archiveli B//native/libhadooppipes.a:current ar archivelib//native/libhadoop.so:symbolic link to ' libhadoop.so.1.0 .0 ' lib//native/libhadoop.so.1.0.0:elf 64-bit LSB shared Object, x86-64, version 1 (SYSV), dynamically linked, not Str Ippedlib//native/libhadooputils.a:current ar archivelib//native/libhdfs.a:current ar archivelib//nati Ve/libhdfs.so:symbolic link to ' libhdfs.so.0.0.0 ' lib//native/libhdfs.so.0.0.0:elf 64-bit LSB shared ob Ject, x86-64, version 1 (SYSV), dynamically linked, not strippedlib//native/libnativetask.a:current ar archivelib/ /native/libnativetask.so:symbolic link to ' libnativetask.so.1.0.0 ' lib//native/libnativetask.so.1.0.0:elf 64-bit LS B shared Object, x86-64, version 1 (SYSV), dynamically linked, not stripped[[email protected] hadoop-3.0.0-snapshot] # pwd/home/fish/hadoop/haDoop-dist/target/hadoop-3.0.0-snapshot 

Compilation issues

Question 1:

[error] Failed to execute goal on project Hadoop-common:could not resolve dependencies for Project Org.apache.hadoop:hadoop-common:jar:3.0.0-snapshot:failure to find org.apache.hadoop:hadoop-auth:jar:tests : 3.0.0-snapshot in Http://10.0.1.88:8081/nexus/content/repositories/thirdparty/was cached in the local repository, Resolution won't be reattempted until the update interval of thirdparty have elapsed or updates are forced-[help 1 ][error][error] To see the full stack trace of the errors, re-run Maven with THE-E switch. [ERROR] Re-run Maven using The-x switch to enable full debug logging. [ERROR] [ERROR] For more information on the errors and possible solutions, please read the following articles:[error] [Help 1] http://c Wiki.apache.org/confluence/display/maven/dependencyresolutionexception[error][error] After correcting the problems , you can resume the build with the Command[error] mvn <goals>-rf:hadoop-common 

Modify the files in the. M2:

mv/root/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0-snapshot/ hadoop-auth-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0- snapshot/hadoop-auth-3.0.0-snapshot-tests.jarmv/root/.m2/repository/org/apache/hadoop/hadoop-kms/3.0.0- snapshot/hadoop-kms-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-kms/ 3.0.0-snapshot/hadoop-kms-3.0.0-snapshot-tests.jarmv/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.0.0- snapshot/hadoop-hdfs-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/ 3.0.0-snapshot/hadoop-hdfs-3.0.0-snapshot-tests.jar

Question 2:

There are some errors can not be downloaded to the jar, this situation can be logged into the http://search.maven.org/official library to see if the package does not exist, if it exists, it may be because of network reasons, more than a few times to do.

Question 3:

[Email protected] bin]#./hadoop:no such file or directory

Modify the format of the Hadoop command for Linux:

Dos2unix/home/fish/hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin/hadoop

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Linux source code Compilation of Hadoop