Hadoop Study Notes (3) -- Build a Distributed Environment

Hadoop Study Notes (3)

-- Build a Distributed Environment

Previously, we have run hadoop on a single machine, but we know that hadoop supports distributed, and its advantages are its outstanding distribution, so we have to simulate it in an environment.

Here, we use this policy to simulate the environment. We use three Ubuntu machines, one as the master machine and the other two as the slave machine (Slaver ). At the same time, we will use the environment set up in Chapter 1 for this host.

We use steps similar to those in Chapter 1:

1. Build running environment

We know that hadoop runs on Linux. So we run it on Ubuntu. Therefore, two slave machines also use the Linux system. In order to save resources, I used two centos systems during the test, and used command lines instead of graphics.

Software preparation in the system. In chapter 1, we have prepared subversion SSH ant and JDK. Here, we don't need to download or compile the code from the host. just copy the code from the host. Therefore, you only need to install SSH and JDK on the slave machine:

Use the sudo apt-Get Install SSH command to install SSH.

Note: In centos, use Yum Install SSH.

Java environment, you can download a JDK installation package online, such as: jdk-6u24-linux-i586.bin

Install and run the./jdk-6u24-linux-i586.bin directly in the directory.

Then configure the JDK directory:

First go to the installation directory CD jdk-6u24 -...

Then enter PWD to view the Java installation directory and copy it:

Command Line execution: sudo gedit/etc/profile

In the open file, append:

Export java_home =/home/Administrator/hadoop/jdk1.6.0 _ 27 // enter the installation directory

Export path =$ {java_home}/bin: $ path

Execute source/etc/profile to take effect immediately

 

1. Network Configuration

To run the distributed environment, the three computers (virtual machines) must be connected to the Internet. At the same time, the three must be unobstructed.

If you directly use a virtual machine, it is more convenient to use the NAT network mode in the virtual machine:

Go to the three systems and run the ifconfig command to check the assigned IP Address:

As shown in, 10.0.0.11 is displayed.

If no eth0 is found, that is, the network card is not enabled or the address is allocated, You can manually allocate the following:

Ifconfig eth0 10.0.0.12 netmask 255.255.255.0 // set the eht0 IP Address

Route add default GW 10.0.0.2 // set the Gateway

In VMware, how can we see the gateway? You can see it in the menu editing> Virtual Network Editor:

The gateway must be configured correctly. Otherwise, the machine cannot Ping each other after the IP address is used.

After configuring the IP address, you can ping the gateway and other machines to see if the IP address is accessible.

Here, the IP addresses of the three servers are:

Host master: 10.0.0.10

Slave machine 1 salter1: 10.0.0.11

Slave machine 2 salter2: 10.0.0.12

 

 

With the IP addresses of three machines, we think we will certainly use the IP addresses later in the configuration, but to facilitate future IP address changes, we should use another name. In window, we know that there is a host file in c: \ windows \ system32 \ driver \ etc. After modification, we can replace the IP address with an alias.

In Linux, this file is also available in/etc/hosts. So EDIT: $ VI/etc/hosts:

After saving the file, you can ping the master to ping node1 instead of the IP address.

This operation must be performed on all three machines.

 

Now that the network is running, the same account must be used on all machines where hadoop is deployed for subsequent operations. Therefore, you need to create an account and password identical to that of the host on two slave machines:

For example, use the zjf account: $ user add zjf to set the password: $ passwd zjf to enter this account: $ su zjf

There may be a firewall on the machine, which affects the remote connection. You can turn it off first:

$ Service iptables stop

 

1. Configure SSH

In chapter 2, we have learned about the SSH function, and it is really useful here.

On the master machine, use SSH to connect to node1:

As you can see, you need to enter the password. It is not a good thing to enter passwords one by one when starting all slave machines remotely. You have to configure the following:

1. Do not enter a password when you log on to node1.

   This is already described in Chapter 1st. I will not talk about it here. The result is:

2. Enable the master node to log on to the two slave nodes (slave) through SSH without a password)

   To achieve this function, the public key file of the two slave nodes must contain the public key information of the master node, so that the master can smoothly and securely access the two slave nodes. The procedure is as follows:

$ Cd ~ /. SSH

$ SCP [email protected]: ~ /. Ssh/id_dsa.pub./master_dsa.pub

$ Cat master_dsa.pub> authorized_keys

 

Okay. After configuration, go back to the master machine and try SSH node1 again:

OK. The password is successfully entered. No Password is required.

Similarly, perform this operation on node2.

 

1. Configure hadoop

Based on the configuration in chapter 1, we need to add two configurations:

In the conf folder, find the Masters file, edit it, enter the master file, and save it:

In the same folder, locate slaves and edit it. Enter node1 node2 and save it:

 

Open the core-site.xml under conf:

Replace localhost with master.

 

Open the marped-site.xml under conf:

In the same way, replace localhost with master.

 

1. Copy A hadoop package

Previously, during the deployment on a machine, our hadoop package downloaded the source code through SVN and then compiled with ant, but here we don't have to worry about it from the machine, we can copy it from the host. How to copy it? We use SSH for remote login, and SCP for remote replication. Before copying, note the locations where hadoop is stored in the host and the slave database.

For example, in the host, we store it under test, so we create the test folder on both nodes.

Then execute: SCP-r hadoop-0.20.2/node1: ~ on the host :~ /Test then you will see a screen refresh, indicating that the screen is being copied.

Same execution: SCP-r hadoop-0.20.2/node2 :~ /Test

 

Now, both slave machines have hadoop packages.

 

1. Run

On the host, go to the hadoop-0.20.2 directory and run the bin/start-all.sh to start the entire distributed system.

Then run JPs on the host:

Run JPs on the slave:

 

Open http: // localhost: 50070 on the master server. You can see:

There are two active nodes. Click in and you can see:

Click the following node to View Details. If the page cannot be opened, the machine firewall may be blocked.

You can enter the corresponding machine and execute

$ Service iptables stop

To disable the firewall.

 

We can try to upload the file:

$ Bin/hadoop FS-put ~ /Tool/eclipse-SDK-3.7.1-linux-gtk.tar.gz test1.tar.gz

You can see:

 

Upload the file again:

$ Bin/hadoop FS-put ~ /Tool/eclipse-SDK-3.7.1-linux-gtk.tar.gz test2.tar.gz

You can see:

 

However, they all ran on the same stage. So you can execute the command

$ Bin/hadoop balancer-threshold 1

In this case, let's take a look:

Balanced.