Install and set Hadoop2.2.0 fully distributed cluster Platform

Prerequisites:
(1) first install JDK 6 or later on each Linux computer, and set JAVA_HOME and so on. Test whether commands such as java, javac, and jps can be used on the terminal, how to configure JDK is not mentioned here;
(2) Install SSH on each Linux server. For how to install SSH on the Linux platform, see Install SSH on the Linux platform. The following describes how to configure SSH logon without a password.

With the preceding prerequisites, we can install the Hadoop distributed platform. The procedure is as follows:

1. First set the IP address of the computer to a static address:

The configuration of static IP addresses varies with Linux releases. Here we will introduce how to set static IP addresses for CentOS, Ubunt, and Fedora 19:
(1) perform the following steps to set a static CentOS IP Address:

[Bkjia @ bkjia hadoop] $ sudo vim/etc/sysconfig/network-scripts/ifcfg-eth0
Add the following statement: IPADDR = 192.168.142.139 NETMASK = 255.255.255.0
NETWORK = 192.168.0.0
Set the IPADDR address you want. Here I am 192.168.142.139.
After setting the IP address, run the following command to make the IP address take effect:

[Bkjia @ bkjia hadoop] $ sudo service network restart Shutting down interface
Eth0: Device state: 3 (disconnected) [OK] Shutting down loopback
Interface: [OK] Bringing up loopback interface: [OK] Bringing up
Interface eth0: Active connection state: activated Active connection
Path:/org/freedesktop/NetworkManager/ActiveConnection/7 [OK] [bkjia @ bkjia
Hadoop] $
Run ifconfig to check whether the settings take effect:
[Bkjia @ bkjia hadoop] $ ifconfig eth0 Link encap: Ethernet HWaddr
00: 0C: 29: 9F: FB: C0 inet addr: 192.168.142.139 Bcast: 192.168.142.255
Mask: 00000000255.0 inet6 addr: fe80: 20c: 29ff: fe9f: fbc0/64 Scope: Link UP
Broadcast running multicast mtu: 1500 Metric: 1 RX packets: 389330 errors: 0
Dropped: 0 overruns: 0 frame: 0 TX packets: 171679 errors: 0 dropped: 0
Overruns: 0 carrier: 0 collisions: 0 txqueuelen: 1000 RX bytes: 473612019
(451.6 MiB) TX bytes: 30110196 (28.7 MiB) lo Link encap: Local Loopback
Inet addr: 127.0.0.1 Mask: 255.0.0.0 inet6 addr ::: 1/128 Scope: Host UP
Loopback running mtu: 16436 Metric: 1 RX packets: 80221 errors: 0 dropped: 0
Overruns: 0 frame: 0 TX packets: 80221 errors: 0 dropped: 0 overruns: 0
Carrier: 0 collisions: 0 txqueuelen: 0 RX bytes: 1051174395 (1002.4 MiB) TX
Bytes: 1051174395 (1002.4 MiB) [bkjia @ bkjia hadoop] $
It can be seen that the IP address has been set to 192.168.142.139!
(2) perform the following steps to set the static IP address of Ubuntu:

Bkjia @ node1 :~ $ Sudo vim/etc/network/interfaces Add: auto eth0 iface
Eth0 inet static address 192.168.142.140 netmask 255.255.255.0 gateway
192.168.142.1
You also need to make the IP address take effect:
Bkjia @ node1 :~ $ Sudo/etc/init. d/networking restart
Also, enter ifconfig to check whether the IP settings take effect.
(3) perform the following steps to set static IP addresses of ora 19 (static IP addresses of other Fedora versions are different from those of version 19, which is not provided here ):

[Bkjia @ bkjia network-scripts] $ sudo vim
/Etc/sysconfig/network-scripts/ifcfg-ens33 to add:
IPADDR0 = 192.168.142.138 NETMASK0 = 255.255.255.0 GATEWAY0 = 192.168.142.0
After setting the IP address, run the following command to make the IP address take effect:
[Bkjia @ bkjia network-scripts] $ sudo service network restart Restarting
Network (via systemctl): [OK]
Also, enter ifconfig to check whether the IP settings take effect.

2. Set the hostname of each host

In step 1, I configured three hosts, CentOS, Ubuntu, and Fedora, respectively. I plan to use them as computers in the cluster, where the Fedora host as the master, and the other two computers as the slave. This step describes how to modify the hostname of the three computers:
(1) perform the following steps to set the hostname of Fedora19:

[Bkjia @ bkjia network-scripts] $ sudo hostnamectl set-hostname master
To check whether the setting takes effect, run the following command [bkjia @ bkjia network-scripts] $ hostname master
(2) Set hostname in Ubuntu as follows:
Bkjia @ node1 :~ $ Sudo vim/etc/hostname Add the hostname you want to obtain in it. Here I will use node1.
To check whether the setting takes effect, run the following command bkjia @ node1 :~ $ Hostname node1
(3) perform the following steps to set the hostname for CentOS:
[Bkjia @ node network-scripts] $ sudo vim/etc/sysconfig/network
Change the HOSTNAME in it to your desired hostname. Here I will use node HOSTNAME = node to check whether the settings take effect. Run the following command
[Bkjia @ node network-scripts] $ hostname node
3. Add the following configurations to/etc/hosts on the three computers:
[Bkjia @ master ~] $ Sudo vim/etc/hosts Add the following statement 192.168.142.138 master
192.168.142.139 node 192.168.142.140 node1
It is actually the correspondence between the static IP addresses of the above three computers and their hostnames. Check whether the modification takes effect. You can use ping to view the modification:
[Bkjia @ master ~] $ Ping node PING node (192.168.142.139) 56 (84) bytes
Data. 64 bytes from node (192.168.142.139): icmp_seq = 1 ttl = 64 time = 0.541.
MS 64 bytes from node (192.168.142.139): icmp_seq = 2 ttl = 64 time = 0.220 MS
^ C --- node ping statistics --- 2 packets transmitted, 2 sent Ed, 0%
Packet loss, time 1000 ms rtt min/avg/max/mdev = 0.220/0.380/0.541/0.161
The MS [bkjia @ master ~] $
If the above command can be pinged, the setting takes effect.

4. Set SSH password-less Login

This blog introduces how to install SSH (Install SSH on Linux), and how to set SSH password-less login ("how to configure SSH for password-less login for Ubuntu and CentOS"). Here we mainly want to talk about precautions, first, after logging on to the master host with SSH without a password, copy the generated id_dsa.pub file to node and node1. Run the following command:
[Bkjia @ localhost ~] $ Cat/home/bkjia/. ssh/id_dsa.pub | \ ssh
Bkjia@192.168.142.139 'cat-> ~ /. Ssh/authorized_keys'
Make sure that the SSH service on the host 192.168.142.139 is running. Bkjia@192.168.142.139's bkjia is the username You need to log on to the host 192.168.142.139. Similarly, you can use a similar command to copy id_dsa.pub to the host 192.168.142.140.
Of course, you can also use the scp command to copy the file to the corresponding host:

[Bkjia @ master Documents] $ scp/home/bkjia/. ssh/id_dsa.pub \
Bkjia@192.168.142.139 :~ /. Ssh/authorized_keys
Check whether you can log on to node and node1 from the master without a password. Run the following command:
[Bkjia @ master Documents] $ ssh node The authenticity of host 'node
(192.168.142.139) 'Can't be established. RSA key fingerprint is
AE: 99: 43: f0: cf: c6: a9: 82: 6c: 93: a1: 65: 54: 70: a6: 97. Are you sure you want
To continue connecting (yes/no )? Yes Warning: Permanently added
'Node, 192.168.142.139 '(RSA) to the list of known hosts. Last login: Wed
Nov 6 14:54:55 2013 from master [bkjia @ node ~] $
The preceding information is displayed when you run the preceding command for the first time. Above [bkjia @ node
~] It implies that we have successfully logged on to the node from the master without a password. If a password is required during the login process, it means that the SSH login without a password is unsuccessful, generally, this is a File Permission issue. For the solution, see how to configure SSH for login without a password in Ubuntu and CentOS.

Download hadoop-2.2.0.tar.gz with the following command:
The following operations are performed on the master machine.

[Bkjia @ bkjia/home] $ mkdir/home/bkjia/Downloads/hadoop [bkjia @ bkjia/home] $ cd
/Home/bkjia/Downloads/hadoop [bkjia @ bkjia hadoop] $ wget \
Http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
After running the command, the hadoop-2.2.0.tar.gz file will be saved in/home/bkjia/Downloads/hadoop. Unzip it.
[Bkjia @ bkjia hadoop] $ tar-zxvf hadoop-2.2.0.tar.gz
The hadoop-2.2.0 folder will be generated under the hadoop folder and the following command will be run
[Bkjia @ bkjia hadoop] $ cd hadoop-2.2.0 [bkjia @ bkjia hadoop-2.2.0] $ ls-l total 56
Drwxr-xr-x. 2 bkjia 4096 Oct 7 bin drwxr-xr-x. 3 bkjia 4096
Oct 7 etc drwxr-xr-x. 2 bkjia 4096 Oct 7 include
Drwxr-xr-x. 3 bkjia 4096 Oct 7 lib drwxr-xr-x. 2 bkjia 4096
Oct 7 libexec-rw-r --. 1 bkjia 15164 Oct 7 LICENSE.txt
Drwxrwxr-x. 3 bkjia 4096 Oct 28 logs-rw-r --. 1 bkjia 101
Oct 7 NOTICE.txt-rw-r --. 1 bkjia 1366 Oct 7 README.txt
Drwxr-xr-x. 2 bkjia 4096 Oct 28 sbin drwxr-xr-x. 4 bkjia 4096
Oct 7 share
The folder of the extracted file is displayed.

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

1 2 Next Page