Build a Hadoop 2.5.1 standalone and pseudo-distributed environment on Ubuntu 14.04 (32-bit)

Build a Hadoop 2.5.1 standalone and pseudo-distributed environment on Ubuntu 14.04 (32-bit)

Introduction

The Ubuntu 32-bit system that has been used all the time (prepare to use Fedora next time, Ubuntu is increasingly unsuitable for Learning). Today we are going to learn about Hadoop, the result is that the latest encapsulated version 2.5.1 released on the Apache official website cannot be started after the configuration is complete. Check the error log and find that the native library version is different from the system version, using the file command, we can find that all the native library files of the encapsulated version are 64-bit:

The version released on the official website began to be compiled in a 64-bit environment (which seems to have been compiled in a 32-bit environment ). No way, you have to download the source code and compile it yourself.

Unexpectedly, all kinds of errors during compilation simply couldn't be stopped, so I had to patiently solve them one by one. During configuration, many configurations were different due to the qualitative changes of the framework introduced since 2.3, online tutorials are really eye-catching... However, it was still forcibly configured and finally ran up. Write it down systematically to avoid future detours.

I mainly referred to two articles (in fact, the integration of the two articles ):

CentOS 6.5 source code compilation and installation Hadoop2.5.1

Hadoop (2.5.1) pseudo-distributed environment CentOS (6.5 64-bit) Configuration

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Environment

Ubuntu 14.04 (32 bit) download: http://www.ubuntu.com/download/desktop

[We recommend that you use a 64-bit system, preferably other Linux releases]

JDK 7u71

[Be sure to use JDK 7 instead of the latest JDK 8. Otherwise, the compilation will fail and the OpenJDK that comes with the system will not be used. Be sure to configure the environment variables so that the default Java environment is JDK 7, be sure to run the "java-version" command to confirm it.] Ant 1.9.4

[Ubuntu official source Ant version 1.9.3 can also be used. If you do not want to manually configure the environment, you can also set "apt-get install ant"] Maven 3.2.3

[Use the latest official version 3.2.3. Do not use version 2 or apt-get. The current version of Ubuntu official source may be too old and may cause compilation failure.] FindBugs 3.0.0

[The latest version 3.0.0 is used. The current version of Ubuntu official source is version 2.0. If it is too old, compilation will fail.] Protobuf 2.5

[Note: The latest official version is version 2.6, but hadoop 2.5.1 requires Protobuf to be version 2.5. Therefore, you must use the 2.5 version provided by the link; otherwise, compilation will fail] Hadoop 2.5.1 src

[For 32-bit systems, compile the source code honestly. Don't go down to the compiled version.]

Note: It is best to keep the above tool versions consistent. using other versions may cause compilation failure. Of course, you can just try it and compile it successfully. After the above tools are correctly installed and configured, the compilation is generally successful.

Compiling environment

Configure the compiling environment first. Install and configure JDK 7u71, Ant 1.9.4, Maven 3.2.3, Findbugs 3.0.0, and Protobuf 2.5 in sequence:

JDK 7u71:

Tar-zxvf jdk-7u71-linux-i586.tar.gz
Sudo cp-r jdk1.7.0 _ 71/usr/lib/jvm/

# Configure environment variables:
Export JAVA_HOME =/usr/lib/jvm // jdk1.7.0 _ 71
Export PATH = $ JAVA_HOME/bin: $ JAVA_HOME/jre // bin: $ PATH
Export CLASSPATH =.: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jar: $ CLASSPATH

# Note:

#1. the java path must be earlier than the original PATH. Otherwise, the java command will be soft connected to the built-in OpenJDK by default. "Java-version" is recommended.

#2. The configuration in/etc/profile,. profile or. bashrc (. zshrc) can be done, but the configuration takes effect only after source .. If bashrc or. zshrc is used, the source will be used once, and/etc/profile or. profile will take effect only once each time the terminal is started until the next boot. Different configuration files have different loading mechanisms.

#3. The configuration of the following tools is the same.

Ant 1.9.4:
Tar zxvf apache-ant-1.9.4-bin.tar.gz

Sudo cp-r apache-ant-1.9.4/usr/share/

# Configure the environment variable: export ANT_HOME =/usr/share/apache-ant-1.9.4 export PATH = $ PATH: $ ANT_HOME/bin

Maven 3.2.3:

Tar-zxvf apache-maven-3.2.3-bin.tar.gz
Sudo cp-r apache-maven-3.2.3/usr/share/

# Configure environment variables:

Export M2_HOME =/usr/share/apache-maven-3.2.3

Export MAVEN_OPTS = "-Xms256m-Xmx512m"
Export PATH = $ PATH: $ M2_HOME/bin

FindBugs 3.0.0:

Tar zxvf findbugs-3.0.0.tar.gz
Udo cp-r findbugs-3.0.0/usr/share/

# Configure environment variables:
Export FINDBUGS_HOME =/usr/share/findbugs-3.0.0
Export PATH = $ PATH: $ FINDBUGS_HOME/bin

Protobuf 2.5.0:

Tar zxvf protobuf-2.5.0.tar.gz

Cd protobuf-2.5.0
./Configure -- prefix =/usr/share
Make

Sudo make install

I installed all the tools in the/usr/share directory. By default, the Ubuntu system will search for the library files in this directory without any other configuration. However, if the tool is installed in the/usr/local directory, Ubuntu will not automatically search for the library files in this directory. In this case, you need to configure the development library environment variable:

Export LD_LIBRARY_PATH =$ {LD_LIBRARY_PATH}:/usr/local/lib

For more details, please continue to read the highlights on the next page:

1 2 3 Next Page