[Spark Asia Pacific Research Institute Series] the path to spark practice-Chapter 1 building a spark cluster (step 2)

Go to the second machine and check the IP address of the Host:

We can see that the IP address of this host is "192.168.184.131 ".

In/etc/hostname, change the host name to "slave1 ":

Save and exit.

To make the modification take effect, restart the machine and check the Host Name:

This indicates that our modification has taken effect.

Go to the third machine and check the IP address of the Host:

We can see that the IP address of this host is "192.168.184.132 ".

In/etc/hostname, change the host name to "slave2"

Save and exit.

To make the modification take effect, restart the machine and check the Host Name:

This indicates that our modification has taken effect.

 

Now, configure the ing between the host name and IP address in/etc/hosts on slave1. After opening:

In this case, we changed the configuration file:

Configure the ing between the host names and IP addresses of "master", "slave1", and "slave2. Save and exit.

Ping the master node to find that there is no problem with network access:

Then, configure the ing between the host name and IP address in/etc/hosts on slave2. The configuration is as follows:

Save and exit.

 

In this case, we can ping both master and slave1;

 

Finally, configure the ing between the host name and IP address in/etc/hosts on the master. The configuration is as follows:

At this time, the ping command is used on the master to communicate with the slave1 and slave2 machines:

It is found that the machines on the two slave nodes have been pinged.

Finally, let's test the communication between the Server Load balancer 1 and master and Server Load balancer 2:

So far, the master, slave1, and slave2 machines have achieved mutual communication!

Step 2: Configure SSH password-less Authentication

First, let's take a look at the situation that the master accesses slave1 through the SSH protocol without Configuration:

We need a password. If you do not log in, log out directly.

How can I enable the cluster to use SSH password-free login?

According to the preceding configuration, a private key id_rsa and a Public Key id_rsa.pub are generated under the/root/. Ssh/directory on the master, slave1, and slave2 machines.

In this case, the id_rsa.pub of slave1 is sent to the master, as shown below:

At the same time, the slave2 id_rsa.pub is sent to the master, as shown below:

Check whether the data has been copied on the master:

Now we can see that the public keys of slave1 and slave2 nodes have been transmitted.

All public keys are integrated on the master node:

Copy the master's public key information authorized_keys to the. SSH directory of slave1 and slave1:

Log on to slave1 and slave2 through SSH again:

At this time, the master does not need a password to log on to slave1 and slave2 through SSH, and the same slave1 or slave2 does not need a password to log on to the other two machines through the SSH protocol.

Step 3: modify the configuration files of master, slave1, and slave2

[Spark Asia Pacific Research Institute Series] the path to spark practice-Chapter 1 building a spark cluster (step 2)