Summary of Hadoop cluster construction on RedHatLinuxAS6

In the home of two computers with VMware + RedHatLinuxAS6 + Hadoop-0.21.0 to build a 3 node Hadoop cluster, although it is already set up a similar cluster, I also ran Java API to operate HDFS and Map/reduce, but this time it was still challenged. Some small details and some omissions would be like a roller coaster. Therefore, write down several steps of this operation for your reference. 1. Create three VM instances to ensure interconnection I

In the home of two computers with VMware + RedHat Linux AS6 + Hadoop-0.21.0 to build a 3 node Hadoop cluster, although it is already set up a similar cluster, I also ran Java API to operate HDFS and Map/reduce, but this time it was still challenged. Some small details and some omissions would be like a roller coaster. Therefore, write down several steps of this operation for your reference.

1. Create three VM instances to ensure interconnection

IP address: 192.168.43.201 hostname: zhangzk01
IP address: 192.168.43.202 hostname: zhangzk02
IP address: 192.168.43.203 hostname: zhangzk03

Take zhangzk01 as the namenode node, and zhangzk02 and zhangzk03 as the datanode.

The/etc/hosts file of the zhangzk01 machine is changed to the following format:

127.0.0.1 localhost. localdomain localhost
192.168.43.201 zhangzk01 zhangzk01.localdomain zhangzk01
192.168.43.202 zhangzk02
192.168.43.203 zhangzk03

The/etc/hosts file of zhangzk02 is changed to the following format:

127.0.0.1 localhost. localdomain localhost
192.168.43.202 zhangzk02 zhangzk02.localdomain zhangzk02
192.168.43.201 zhangzk01

192.168.43.203 zhangzk03

The/etc/hosts file of zhangzk03 is changed to the following format:

127.0.0.1 localhost. localdomain localhost
192.168.43.203 zhangzk03 zhangzk03.localdomain zhangzk03
192.168.43.201 zhangzk01

192.168.43.202 zhangzk02

2. Ensure that the Master/Slave machines can access each other through SSH without a key

Create the same user on the three machines, zhangzk. The home directory is/home/zhangzk.

Zhangzk01:

Log on to zhangzk01 as a user, and run the following command in the home directory:

Ssh-keygen-t rsa

Press enter to create two files, id_rsa.pub and id_rsa, In the/home/zhangzk/. ssh/directory.

Next, enter the/home/zhangzk/. ssh directory and execute the following command:

Cd. ssh

Copy the is_rsa.pub file to the zhangzk02 and zhangzk03 machines.

Scp-r id_rsa.pub zhangzk @ zhangzk03:/home/zhangzk/. ssh/authorized_keys_zhangzk01
Scp-r id_rsa.pub zhangzk @ zhangzk02:/home/zhangzk/. ssh/authorized_keys_zhangzk01

Zhangzk02:

Log on to zhangzk02 as the user zhangzk and run the following command in the directory:

Ssh-keygen-t rsa

Press enter to create two files, id_rsa.pub and id_rsa, In the/home/zhangzk/. ssh/directory.

Next, enter the/home/zhangzk/. ssh directory and execute the following command:

Cd. ssh

Copy the is_rsa.pub file to zhangzk01.

Scp-r id_rsa.pub zhangzk @ zhangzk01:/home/zhangzk/. ssh/authorized_keys_zhangzk02

Zhangzk03:

Log on to zhangzk03 using the user zhangzk and run the following command in the directory:

Ssh-keygen-t rsa

Press enter to create two files, id_rsa.pub and id_rsa, In the/home/zhangzk/. ssh/directory.

Next, enter the/home/zhangzk/. ssh directory and execute the following command:

Cd. ssh

Copy the is_rsa.pub file to zhangzk01.

Scp-r id_rsa.pub zhangzk @ zhangzk01:/home/zhangzk/. ssh/authorized_keys_zhangzk03

**************************************** ***************************************

The above method generates an rsa key for the zhangzk01 \ zhangzk02 \ zhangzk03 machine, copies the token of zhangzk01 to zhangzk02 \ zhangzk03, and copies the token of zhangzk02 and zhangzk03 to zhangzk01.

Next, complete the following steps:

Zhangzk01:

Log on to zhangzk01 as a zhangzk user and go to the/home/zhangzk/. ssh directory. Run the following command:

Cat id_rsa.pub> authorized_keys
Cat authorized_keys_zhangzk02> authorized_keys
Cat authorized_keys_zhangzk03> authorized_keys
Chmod 644 authorized_keys

Zhangzk02:

Log on to zhangzk02 as a zhangzk user and go to the/home/zhangzk/. ssh directory. Run the following command:

Cat id_rsa.pub> authorized_keys
Cat authorized_keys_zhangzk01> authorized_keys
Chmod 644 authorized_keys

Zhangzk03:

Log on to zhangzk03 as a zhangzk user and go to the/home/zhangzk/. ssh directory. Run the following command:

Cat id_rsa.pub> authorized_keys
Cat authorized_keys_zhangzk01> authorized_keys
Chmod 644 authorized_keys

Through the above configuration, you can access zhangzk02 and zhangzk03 by logging on to zhangzk without key authentication. You can also connect to zhangzk01 by Using ssh zhangzk01 on zhangzk02 and zhangzk03.