[Raspberry Pi3] Hadoop build and configuration

Causion:

Only a raspberry Pi3 to play, the bottleneck is not in the CPU, 1 g of memory is too little, running a hadoop is very hard

Here is the process of the blind toss:

The installation process for Oracle's arm JDK has been skipped right here.

1. Download the source code for Hadoop,

Because raspberry is a 64bit CPU but actually runs a 32-bit system, there is no direct use of the binary, this way to download and compile the source code, in order to improve the efficiency of the next run, reduce the amount of memory required.

This download version is 2.7.2, the download is done after the decompression

2. Install the necessary software, you can refer to the source code directory under the BUILDING.txt

*Maven $sudoApt-get-yInstallmaven*Native Libraries $sudoApt-get-yInstallBuild-essential autoconf automake libtool cmake Zlib1g-devPkg-configlibssl-Dev* Protocolbuffer2.5.0(required) $sudoApt-get-yInstallLibprotobuf-dev protobuf-compileroptional Packages:*Snappy Compression $sudoApt-getInstallSnappy libsnappy-Dev*BZIP2 $sudoApt-getInstall bzip2libbz2-Dev* Jansson (C Library forJSON) $sudoApt-getInstalllibjansson-Dev*Linux FUSE $sudoApt-getInstallFuse Libfuse-dev

3. Install the MAVEN plugin for Hadoop

CD hadoop-maven-plugins/mvn install

4. Because the version of Protobug that is used on the system is 2.6+, you need to modify the version of the protobuf that is required by default here.

Otherwise the error will be reported as follows:

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7. 2 ' Libprotoc 2.6.1 ' ' 2.5.0 ' 1]

To modify a file:

Hadoop-project/pom.xml

Modify the version to

<protobuf.version>2.6. 1</protobuf.version>

5. Lay the arm hard-float patch, otherwise compile native code will have soft float compile problem

The error is as follows:

[EXEC] soft-floatJVM detected [exec]--looking forexit [exec]--looking forExit-Not found [exec]--Configuring incomplete, Errors occurred![exec] See also"/mnt/timemachine/hadoop/hadoop-2.7.2-src/hcmake Error at jniflags.cmake:59 (message):[EXEC] soft-floatDev libraries required (e.g.'apt-get Install Libc6-dev-armel'[exec] on Deadoop-common-project/hadoop-common/target/native/cmakefiles/cmakeoutput.log".[EXEC] See Also"/mnt/timemachine/hadoop/hadoop-2.7.2-srcbian/ubuntu)

Patch with the following command

CD hadoop-common-project/hadoop-common/srcwget https://issues.apache.org/jira/ Secure/attachment/12570212/hadoop-9320.patchPatch < hadoop-9320. Patch

6. Compile the source code

MVN compile-pnative

7. After the compilation OK, packaging, do not do one of the test links, less memory, can't play

MVN  package-pnative-dtar-dskiptests

The package will be placed in the following position, copied out of the decompression, the configuration will be able to use the

8. The next step is the configuration and start-up.

Check under native Lib is it all right?

bin/Hadoop org.apache.hadoop.util.NativeLibraryChecker -/ ,/ - Ten: -:xxINFObzip2. Bzip2factory:successfully Loaded & Initialized native-bzip2Library system-native -/ ,/ - Ten: -:xxINFO zlib. Zlibfactory:successfully Loaded & Initialized native-zlib librarynative Library Checking:hadoop:true/mnt/timemachine/hadoop/hadoop-2.7.2/lib/native/libhadoop.so.1.0.0zlib:true/lib/arm-linux-gnueabihf/libz.so.1Snappy:true/usr/lib/libsnappy.so.1lz4:trueRevision Aboutbzip2:true/lib/arm-linux-gnueabihf/libbz2.so.1OpenSSL:true/usr/lib/arm-linux-gnueabihf/libcrypto.so

The native library has a

I also only have one, so also can only with a pseudo-cluster, namenode and Datanode are all equipped with the same pi, the next operation and configuration steps are directly referring to the official online instructions can be

Http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

[Raspberry Pi3] Hadoop build and configuration