Build Hadoop2.6 Cluster Environment (hdfs ha + YARN)

Source: Internet
Author: User

Build Hadoop2.6 Cluster Environment (hdfs ha + YARN)

Hadoop2.6 Cluster Environment (hdfs ha + YARN) the original 4G memory can also be willful once.

Preparations:

1. 4 GB memory for the notebook, OS WIN7 (diaosi configuration)

2. VMware Workstation

3. virtual machines: four in CentOS6.4

Virtual Machine Settings:

Each machine: Memory 512 M, hard disk 40G, network adapter: NAT Mode

 

Select advanced to generate a new Virtual Machine Mac address (clone the virtual machine, the Mac address will not change, and then manually generate each time)

 

Edit Virtual Machine network:

 

Click "NAT Settings" to view the Virtual Machine gateway IP address and remember it. this IP address is very important in the Virtual Machine network settings.

The default nat ip address is automatically generated, but the IP address in our cluster needs to be set manually.

 

Local Win7: VMnet8 network settings

 

Lab environment:

Ip

Hostname

Role

192.168.249.130

SY-0130

ActiveNameNode

192.168.249.131

SY-0131

StandByNameNode

192.168.249.132

SY-0132

DataNode1

192.168.249.20.

SY-0133

DataNode2

 

Linux network settings:

1. Create a user, such as hadoop. It is not recommended to use the root user to build a cluster (the root permission is too large)

2. Grant sudo permissions to hadoop users.

 
[root@SY-0130 ~]# vi /etc/sudoers## Allow root to run any commands anywhereroot    ALL=(ALL)       ALLhadoop  ALL=(ALL)       ALL
 

3. view the NIC device used by the current VM

[root@SY-0130 hadoop]# ifconfigeth2      Link encap:Ethernet  HWaddr 00:50:56:35:8E:E8            inet addr:192.168.249.130  Bcast:192.168.249.255  Mask:255.255.255.0          inet6 addr: fe80::250:56ff:fe35:8ee8/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:877059 errors:0 dropped:0 overruns:0 frame:0          TX packets:597769 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:865720294 (825.6 MiB)  TX bytes:324530557 (309.4 MiB)          Interrupt:19 Base address:0x2024 lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host          UP LOOPBACK RUNNING  MTU:16436  Metric:1          RX packets:1354 errors:0 dropped:0 overruns:0 frame:0          TX packets:1354 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:196675 (192.0 KiB)  TX bytes:196675 (192.0 KiB)
 

[Root @ SY-0130 ~] # Cat/proc/net/dev, current Nic, my ID is eth2

[root@SY-0130 ~]# cat /proc/net/devInter-|   Receive                                                |  Transmit face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed    lo:  196675    1354    0    0    0     0          0         0   196675    1354    0    0    0     0       0          0  eth2:865576893  875205    0    0    0     0          0         0 324425517  596433    0    0    0     0       0          0

 

4. view the Mac address of the current Nic

 
[root@SY-0130 ~]# cat /etc/udev/rules.d/70-persistent-net.rules # This file was automatically generated by the /lib/udev/write_net_rules# program, run by the persistent-net-generator.rules rules file.## You can modify it, as long as you keep each rule on a single# line, and change only the value of the NAME= key.# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:b5:fd:bb", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"# PCI device 0x1022:0x2000 (vmxnet)SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:35:8e:e8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

5. Network Configuration

[Root @ SY-0130 ~] # Setup

 

 

Select eth2, set it, change it to eth2, and set the IP address, network management, and DNS.

 

The DNS Server is consistent with the DNS Server in the Win7 network, so that the virtual machine can also connect to the Internet to facilitate the download and installation of software.

Disable the firewall.

6. Modify the hostname

[root@SY-0130 ~]# vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=SY-0130

7. Modify hosts

[Hadoop @ SY-0130 ~] $ Sudo vi/etc/hosts

# Add the following content

192.168.249.130 SY-0130192.168.249.131 SY-0131192.168.249.132 SY-0132192.168.249.133 SY-0133

8. Restart VM. reboot

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

For more details, please continue to read the highlights on the next page:

  • 1
  • 2
  • Next Page

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.