Virtual machine to build Hadoop all distributed cluster-in detail (4)

Virtual machine to build Hadoop all distributed cluster-in detail (1)

Virtual machine to build Hadoop all distributed cluster-in detail (2)

Virtual machine to build Hadoop all distributed cluster-in detail (3)

In the above three blogs, three CentOS6.5 virtual machines were created on the VirtualBox, as master and SLAVE1 nodes of a fully distributed Hadoop cluster as well as SLAVE2 nodes, and set up SSH password-free logins between three virtual machines. Install the JDK on three virtual machines and set the environment variables. So the preparation of the whole Hadoop distributed cluster is complete. Three virtual machines are as follows

	Virtual machine Name	Host Name	IP Address
Master Node	Centos6.5-master	Master	192.168.1.110
SLAVE1 node	Centos6.5-slave1	Slave1	192.168.1.111
Slave2 node	Centos6.5-slave2	Slave2	192.168.1.112

The master and slave nodes of Hadoop are deployed on three virtual machines below. An ordinary user named Sunnie has been created on three virtual machines before. Deployment and configuration Process reference http://www.linuxidc.com/Linux/2014-05/101687.htm

Hadoop Download Address hadoop-1.2.1.tar.gz

Log on to host Master as a user Sunnie.

Copy the downloaded hadoop-1.2.1.tar.gz to the/home/sunnie/documents/directory and unzip

Create a new directory tmp under/HOME/SUNNIE/DOCUMENTS/HADOOP-1.2.1/

Next, configure hadoop-env.sh, Core-site.xml, Hdfs-site.xml, Mapred-site.xml, masters, and slaves files separately.

vim/home/sunnie/documents/hadoop-1.2.1/conf/hadoop-env.sh

Add export java_home=/usr/java/jdk1.8.0_05 at the end

Vim/home/sunnie/documents/hadoop-1.2.1/conf/core-site.xml

Configure the IP and port of name node in Core-site.xml, set the value of Fs.default.name, and change to the following

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://Master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/sunnie/Documents/hadoop-1.2.1/tmp</value>
</property>
</configuration>

Next Modify Hdfs-site.xml

Vim/home/sunnie/documents/hadoop-1.2.1/conf/hdfs-site.xml

Change to the following content

<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

Modify Mapred-site.xml Below

Vim/home/sunnie/documents/hadoop-1.2.1/conf/mapred-site.xml

Configure the IP and port of the job tracker in Mapred-site.xml, set the Mapred.job.tracker value, and change to the following

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>Master:9001</value>
</property>
</configuration>

Configure Masters and slaves separately below

Vim/home/sunnie/documents/hadoop-1.2.1/conf/masters

Remove the localhost from the file and replace it with

Master

Vim/home/sunnie/documents/hadoop-1.2.1/conf/slaves

Remove the localhost from the file and replace it with

Slave1

Slave2

In this way, the Hadoop on host master is configured, and the next thing to do is to copy hadoop-1.2.1 from master to Slave1 and Slave2, respectively.

Log on as user Sunnie first on host Slave1 and host Slave2

Executing on host Master

Scp-r/home/sunnie/documents/hadoop-1.2.1 sunnie@slave1:/home/sunnie/documents

Scp-r/home/sunnie/documents/hadoop-1.2.1 sunnie@slave2:/home/sunnie/documents

At this point, the deployment of the Hadoop distributed cluster on the three virtual machines is complete, below to test the ~

First, shut down the firewall as root on three hosts respectively

Chkconfig iptables off

Then switch back to user Sunnie on all three hosts

Executing on the master host

/home/sunnie/documents/hadoop-1.2.1/bin/hadoop Namenode-format

/home/sunnie/documents/hadoop-1.2.1/bin/start-all.sh

Then execute the JPS command on host master to see

Then execute the JPS command on the Slave1 to see

Performing JPS on the Slave2, you can see

Finally execute on Master

/home/sunnie/documents/hadoop-1.2.1/bin/stop-all.sh

Can see

At this point, with three CentOS6.5 virtual computers deployed Hadoop1.2.1 full distributed cluster is done ~