Linux source code compilation for Hadoop

Hadoop begins, as is customary. Compile the source code first. Import to eclipse. In this way, we need to know about that piece, or it's a problem. Find the source code directly.

before compiling the hadoop2.4.1 source code. Maven and ant environments must be installed, and Hadoop needs protoc2.5.0 support, so download Protoc as well. I downloaded:protobuf-2.5.0.tar.bz2

There are several dependencies to install before compiling the PROTOC: Gcc,gcc-c++,make assumes that you have installed the ability to ignore

Yum Install Gccyum install Gcc-c++yum install Makeyum install cmake  yum install openssl-devel  yum install ncurses -devel  

Installing PROTOC

TAR-XVF protobuf-2.5.0.tar.bz2  cd protobuf-2.5.0  ./configure--prefix=/opt/protoc/make  && make Install

The Linux system runs the compile command:mvn install Eclipse:eclipse-pdist,native-dskiptests-dtar -dmaven.javadoc.skip=true

After the compilation is complete. To view the Hadoop-dist directory:

[Email protected] target]# lltotal 153824drwxr-xr-x. 2 root root      4096 Jul  9 17:00 antrun-rw-r--r--. 1 root root      4809 Jul  9 17:00 dist-layout-stitching.sh-rw-r --r--. 1 root root       666 Jul  9 17:01 dist-tar-stitching.shdrwxr-xr-x. 9 root root      4096 Jul  9 17:00 hadoop-3.0.0-sn Apshot-rw-r--r--. 1 root root 157482988 Jul  9 17:01 hadoop-3.0.0-snapshot.tar.gz-rw-r--r--. 1 root root      3445 Jul  9 17:01 Hadoop -dist-3.0.0-snapshot.jardrwxr-xr-x. 2 root root      4096 Jul  9 17:01 maven-archiverdrwxr-xr-x. 2 root root      4096 Jul  9 17:00 test-dir[[email PR Otected] target]# Pwd/home/fish/hadoop/hadoop-dist/target

to view the version number of Hadoop:

[Email protected] bin]# Cd/home/fish/hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin[[email protected] bin]#. Hadoop versionhadoop 3.0.0-snapshotsource Code repository Https://github.com/apache/hadoop.git-r E0febce0e74ec69597376774f771da46834c42b1compiled by Root on 2015-07-09t08:53zcompiled with Protoc 2.5.0From source with Checksum d69dd13fde158d22d95a263a0f12bc8this command was run using/home/fish/hadoop/hadoop-dist/target/ Hadoop-3.0.0-snapshot/share/hadoop/common/hadoop-common-3.0.0-snapshot.jar[[email protected] bin]# pwd/home/fish/ Hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin

To view some information about the compilation:

[[email protected] hadoop-3.0.0-snapshot]# file lib//native/*lib//native/libhadoop.a:current ar archiveli B//native/libhadooppipes.a:current ar archivelib//native/libhadoop.so:symbolic link to ' libhadoop.so.1.0 .0 ' lib//native/libhadoop.so.1.0.0:elf 64-bit LSB shared Object, x86-64, version 1 (SYSV), dynamically linked, not Str Ippedlib//native/libhadooputils.a:current ar archivelib//native/libhdfs.a:current ar archivelib//nati Ve/libhdfs.so:symbolic link to ' libhdfs.so.0.0.0 ' lib//native/libhdfs.so.0.0.0:elf 64-bit LSB shared ob Ject, x86-64, version 1 (SYSV), dynamically linked, not strippedlib//native/libnativetask.a:current ar archivelib/ /native/libnativetask.so:symbolic link to ' libnativetask.so.1.0.0 ' lib//native/libnativetask.so.1.0.0:elf 64-bit LS B shared Object, x86-64, version 1 (SYSV), dynamically linked, not stripped[[email protected] hadoop-3.0.0-snapshot] # pwd/home/fish/hadoop/haDoop-dist/target/hadoop-3.0.0-snapshot 

Compilation issues

Question 1:

[error] Failed to execute goal on project Hadoop-common:could not resolve dependencies for Project Org.apache.hadoop:hadoop-common:jar:3.0.0-snapshot:failure to find org.apache.hadoop:hadoop-auth:jar:tests : 3.0.0-snapshot in Http://10.0.1.88:8081/nexus/content/repositories/thirdparty/was cached in the local repository, Resolution won't be reattempted until the update interval of thirdparty have elapsed or updates are forced-[help 1 ][error][error] To see the full stack trace of the errors, re-run Maven with THE-E switch. [ERROR] Re-run Maven using The-x switch to enable full debug logging. [ERROR] [ERROR] For more information on the errors and possible solutions, please read the following articles:[error] [Help 1] http://c Wiki.apache.org/confluence/display/maven/dependencyresolutionexception[error][error] After correcting the problems , you can resume the build with the Command[error] mvn <goals>-rf:hadoop-common 

Change the file in the. M2:

mv/root/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0-snapshot/ hadoop-auth-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0- snapshot/hadoop-auth-3.0.0-snapshot-tests.jarmv/root/.m2/repository/org/apache/hadoop/hadoop-kms/3.0.0- snapshot/hadoop-kms-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-kms/ 3.0.0-snapshot/hadoop-kms-3.0.0-snapshot-tests.jarmv/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/3.0.0- snapshot/hadoop-hdfs-3.0.0-snapshot-tests.jar.lastupdated/root/.m2/repository/org/apache/hadoop/hadoop-hdfs/ 3.0.0-snapshot/hadoop-hdfs-3.0.0-snapshot-tests.jar

Question 2:

There are some errors can not be downloaded to the jar, such a situation can be logged into the http://search.maven.org/official library to see if the package does not exist, if it exists, it may be due to network reasons, run more than a few times.

Question 3:

[Email protected] bin]#./hadoop:no such file or directory

To change the format of the Hadoop command for Linux:

Dos2unix/home/fish/hadoop/hadoop-dist/target/hadoop-3.0.0-snapshot/bin/hadoop

Linux source code compilation for Hadoop