Build Hadoop2.6 Cluster Environment (hdfs ha + YARN)

Build Hadoop2.6 Cluster Environment (hdfs ha + YARN)

Hadoop2.6 Cluster Environment (hdfs ha + YARN) the original 4G memory can also be willful once.

Preparations:

1. 4 GB memory for the notebook, OS WIN7 (diaosi configuration)

2. VMware Workstation

3. virtual machines: four in CentOS6.4

Virtual Machine Settings:

Each machine: Memory 512 M, hard disk 40G, network adapter: NAT Mode

 

Select advanced to generate a new Virtual Machine Mac address (clone the virtual machine, the Mac address will not change, and then manually generate each time)

 

Edit Virtual Machine network:

 

Click "NAT Settings" to view the Virtual Machine gateway IP address and remember it. this IP address is very important in the Virtual Machine network settings.

The default nat ip address is automatically generated, but the IP address in our cluster needs to be set manually.

 

Local Win7: VMnet8 network settings

 

Lab environment:

Ip	Hostname	Role
192.168.249.130	SY-0130	ActiveNameNode
192.168.249.131	SY-0131	StandByNameNode
192.168.249.132	SY-0132	DataNode1
192.168.249.20.	SY-0133	DataNode2

 

Linux network settings:

1. Create a user, such as hadoop. It is not recommended to use the root user to build a cluster (the root permission is too large)

2. Grant sudo permissions to hadoop users.

 

[root@SY-0130 ~]# vi /etc/sudoers## Allow root to run any commands anywhereroot    ALL=(ALL)       ALLhadoop  ALL=(ALL)       ALL

 

3. view the NIC device used by the current VM

[root@SY-0130 hadoop]# ifconfigeth2      Link encap:Ethernet  HWaddr 00:50:56:35:8E:E8            inet addr:192.168.249.130  Bcast:192.168.249.255  Mask:255.255.255.0          inet6 addr: fe80::250:56ff:fe35:8ee8/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:877059 errors:0 dropped:0 overruns:0 frame:0          TX packets:597769 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:865720294 (825.6 MiB)  TX bytes:324530557 (309.4 MiB)          Interrupt:19 Base address:0x2024 lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host          UP LOOPBACK RUNNING  MTU:16436  Metric:1          RX packets:1354 errors:0 dropped:0 overruns:0 frame:0          TX packets:1354 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:196675 (192.0 KiB)  TX bytes:196675 (192.0 KiB)

 

[Root @ SY-0130 ~] # Cat/proc/net/dev, current Nic, my ID is eth2

[root@SY-0130 ~]# cat /proc/net/devInter-|   Receive                                                |  Transmit face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed    lo:  196675    1354    0    0    0     0          0         0   196675    1354    0    0    0     0       0          0  eth2:865576893  875205    0    0    0     0          0         0 324425517  596433    0    0    0     0       0          0

 

4. view the Mac address of the current Nic

 

[root@SY-0130 ~]# cat /etc/udev/rules.d/70-persistent-net.rules # This file was automatically generated by the /lib/udev/write_net_rules# program, run by the persistent-net-generator.rules rules file.## You can modify it, as long as you keep each rule on a single# line, and change only the value of the NAME= key.# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:b5:fd:bb", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:35:8e:e8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

5. Network Configuration

[Root @ SY-0130 ~] # Setup

 

 

Select eth2, set it, change it to eth2, and set the IP address, network management, and DNS.

 

The DNS Server is consistent with the DNS Server in the Win7 network, so that the virtual machine can also connect to the Internet to facilitate the download and installation of software.

Disable the firewall.

6. Modify the hostname

[root@SY-0130 ~]# vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=SY-0130

7. Modify hosts

[Hadoop @ SY-0130 ~] $ Sudo vi/etc/hosts

# Add the following content

192.168.249.130 SY-0130192.168.249.131 SY-0131192.168.249.132 SY-0132192.168.249.133 SY-0133

8. Restart VM. reboot

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

For more details, please continue to read the highlights on the next page:

1 2 Next Page