Build a Hadoop cluster tips (1)

1 installation version

Build Hadoop2.4.0 version, based on Ubuntu12.04 x86_64, jdk1.7.0_79

2 References:

1) Reliable Installation documentation http://www.aboutyun.com/thread-7684-1-1.html

2) Official Installation documentation Http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/ClusterSetup.html#Installation

3 Main ideas:

The basic idea of a fully distributed cluster is the process of establishing SSH without password communication, installing JDK, installing Hadoop, testing Hadoop and so on.

First, in Linux-based systems, user rights are relatively error-prone places, the specific settings can refer to Aboutyun Forum related literature.

All operations in this article are done under root, and the actual environment is recommended to be configured under other user rights.

4 SSH no password interconnect:

SSH password-free login principle is to share the public key, the host of the mutual public key can be the SSH protocol without password login.

You need to configure the/etc/hostname file and the/etc/hosts file before generating the public key

The clusters built here are:

10.174.72.4	Root/123	Master
10.174.72.5	Root/123	Slave1
10.174.72.6	Root/123	Slave2
10.174.72.7	Root/123	Slave3
10.174.72.8	Root/123	Slave4

Indicates IP address, user/password, host name, respectively

To modify the host name by configuring the/etc/hostname file

[Email protected]:/# vi/etc/hostname

Add IP address to host name mappings by configuring the/etc/hosts file

[Email protected]:/# vi/etc/hosts

Modify the above two files on each slave node

Summary of Common Linux commands

The specific steps to set up SSH are as follows:

First step: Generate the key

[Email protected]:/# ssh-keygen-t RSA

Press the ENTER key until the end

Step two: Enter the SSH directory and write the public key

[Email protected]:/# cd ~/.ssh/[email protected]:~/.ssh# cat Id_rsa.pub >>authorized_keys

You can login localhost without a password at this time

[Email protected]:~/.ssh# ssh localhost

Step three: Set the remote password-free login to send the public key to slave1 through the SCP command

[Email protected]:~/.ssh# cat id_rsa.pub >> master_key [email protected]:~/.ssh# SCP master_key [email Pro tected]:/root/.ssh/

Write Authorized_keys

[Email protected]:~/.ssh# cat Master_key >> Authorized_keys

"Note": Each of the two machines completes the public key write operation

5 Main Ideas Install JDK

Detailed installation steps reference how to install the Oracle Java JDK on Ubuntu Linux

Install compiled software in Ubuntu, the general step is to unzip the installation package, modify the configuration file, set the PATH environment variable

Unzip the JDK

[Email protected]:/usr/local# tar-zxvf jdk-7u79-linux-x64.tar.gz

Set path

[Email protected]:/usr/local# vi/etc/profile

Add at the bottom of the file

Export java_home=/usr/local/java/jdk1.7. 0_79export CLASSPATH=.: $JAVA _home/lib: $JAVA _home/jre/  Libexport PATH= $PATH: $JAVA _home/bin

Checking syntax with the source command

[Email protected]:/usr/local# source/etc/profile

If you need to compile the source file in the console, you need to modify the CLASSPATH variable, this does not introduce

Build a Hadoop cluster tips (1)