Build a Hadoop cluster on Ubuntu

1. Install JDKa) download the JDK Installation File jdk-6u30-linux-i586.bin under Linux from here. B) copy the JDK installation file to a local directory and select the/opt directory. C) & 1. Install JDK

A) download the JDK Installation File jdk-6u30-linux-i586.bin under Linux from here.

B) copy the JDK installation file to a local directory and select the/opt directory.

C) Execution

Sudo sh jdk-6u30-linux-i586.bin (if you cannot execute chmod + x jdk-6u30-linux-i586.bin first)

D) after installation, modify the/etc/environment file,

Sudo vi/etc/environment

Add

PATH = "...... : /Bin"

CLASSPATH = ".: /Lib"

JAVA_HOME =" "

E) then

Source ~ /. Bashrc

F) Restart terminal. If it is an ssh logon, log out first and then log on again.

G) java-version

You can see the installed jdk information.

Install JDK on the machines in each cluster as shown above.

Related reading:

Hadoop MapReduce data stream http://www.linuxidc.com/Linux/2012-02/54485.htm

Hadoop-based storage platform solves the difficulties of http://www.linuxidc.com/Linux/2012-02/54267.htm in Ocean Data Islands

Performance analysis http://www.linuxidc.com/Linux/2012-02/53821.htm of Hadoop File System in model and architecture

Hadoop cluster beginner's note http://www.linuxidc.com/Linux/2012-02/53524.htm

2. Create a hadoop user on each machine in the cluster.

A) sudo adduser -- ingroup adminhadoop

Password and other information.

Add the new hadoop user to the admin user group, so that the user has the sudo permission, which is convenient During setup.

If a new user error is found, you can use

Userdel-r

Command to delete a user.

B) copy the hadoop-0.installation file hadoop-0.511203.0rc1.tar.gz TO THE hadoop user's home directory of each machine and execute

Tar xzf hadoop-0.20.203.0rc1.tar.gz

Unzip the hadoop-0.20.203.0 directory.

 

3. Configure ssh so that the master machine can access itself and other machines without a password.

A) This step is run on the final master machine. Make sure that each machine has an ssh server and client installed. If not, run

Run the sudo apt-get install ssh and sudo apt-get installopenssh-server commands for installation.

B) first execute the command

Ssh-keygen-t rsa (the system prompts you to directly select the default path for the input path. The password is blank)

Form public/private key pairs and store them in ~ /. Ssh/id_rsa.pub and ~ /. Ssh/id_rsa.

First, copy id_rsa.pub to the local ~ /. Ssh/authorized_keys.

If the original authorized_keys already exists, run

Cat ~ /. Ssh/id_rsa.pub> ~ /. Ssh/authorized_keys: append the public key to authorized_keys.

If the file does not exist, copy id_rsa.pub directly.

Cp ~ /. Ssh/id_rsa.pub ~ /. Ssh/authorized_keys

In this way, access the local machine through ssh

Ssh localhost

No Password is required.

C) add the Public Key id_rsa.pub generated on the master machine to the authorized_keys of other machines. If authorized_keys does not exist, copy it. If authorized_keys exists, append the public key. (Sometimes it does not exist on other machines ~ /. Ssh directory. You can create this directory manually or log on to another machine using ssh ).

D) Finally, you can try to log on to hadoop users on the master machine without a password.