Hadoop 2.7.1 high-availability installation and deployment

Source: Internet
Author: User

Hadoop 2.7.1 high-availability installation and deployment
Hadoop cluster planning goals: create two NameNode for high availability. One NameNode fails and the other can be started. One runs Yarn, three DataNode, and three Zookeeper clusters for high availability. Hadoop2 usually consists of two NameNode, one in active state and the other in standby state. Active NameNode provides external services, while Standby NameNode does not provide external services. It only synchronizes the status of active namenode, so that it can quickly switch when it fails. A zookeeper cluster is also configured for ZKFC (DFSZKFailoverController) failover. When Active NameNode fails, the Standby NameNode status is automatically switched to standby status.
Install hadoop and Zookeeper in/soft;

 

Http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

0. Install the CentOS6.5 and 64-bit operating systems in the system environment with minimal installation. To achieve the goal, use the VMware Virtual Machine to build 6 servers. Therefore, we recommend that the servers have at least 8 GB of memory; vmware uses VMware.®Workstation 11.0.0 build-2305329; the network configuration is as follows:
The virtual machine configuration is as follows: 1. Synchronize the Machine Time
  1. Yum install-y ntp # install Time Service
  2. Ntpdate us.pool.ntp.org # synchronization time
2. Set the Host IP Address
  1. vi /etc/sysconfig/network-scripts/ifcfg-eth0
  1. DEVICE=eth0
  2. IPADDR=192.168.8.101
  3. NETMASK=255.255.255.0
  4. GATEWAY=192.168.8.2
  5. HWADDR=00:0C:29:56:63:A1
  6. TYPE=Ethernet
  7. UUID=ecb7f947-8a93-488c-a118-ffb011421cac
  8. ONBOOT=yes
  9. NM_CONTROLLED=yes
  10. BOOTPROTO=none
Then restart the network service.
  1. service network restart
View ip configuration
  1. ifconfig eth0
  1. eth0 Link encap:EthernetHWaddr00:0C:29:6C:20:2B
  2. inet addr:192.168.8.101Bcast:192.168.8.255Mask:255.255.255.0
  3. inet6 addr: fe80::20c:29ff:fe6c:202b/64Scope:Link
  4. UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
  5. RX packets:777 errors:0 dropped:0 overruns:0 frame:0
  6. TX packets:316 errors:0 dropped:0 overruns:0 carrier:0
  7. collisions:0 txqueuelen:1000
  8. RX bytes:70611(68.9KiB) TX bytes:49955(48.7KiB)
This indicates that the IP address we configured is successful. Note: five virtual machines are cloned using vmware, and an error occurs when configuring the NIC: "Bring up interface eth0: device eth0 does not seem to be present, delaying initialization "solution: Step 1 delete the file 70-persistent-net.rules
  1. rm -f /etc/udev/rules.d/70-persistent-net.rule
Step 2 modify ifcfg-eth0
  1. vim /etc/sysconfig/network-scripts/ifcfg-eth0
Step 3: restart the server
  1. reboot
In this way, you can set a new IP address. 3. Set the host name to hadoop01.
  1. vi /etc/sysconfig/network
Modify the hostname:
  1. NETWORKING=yes
  2. HOSTNAME=hadoop01
  3. NETWORKING_IPV6=no
To modify hosts
  1. vi /etc/hosts
  1. 127.0.0.1 localhost
  2. 192.168.8.101 hadoop01 192.168.8.102 hadoop02 192.168.8.103 hadoop03 192.168.8.104 hadoop04 192.168.8.105 hadoop05 192.168.8.106 hadoop06

Disable ipv6

1. Check whether ipv6 is enabled in the system.

A) view the ENI attributes

Command: ifconfig

Note: "inet6 addr:..." is available :....... The ipv6 feature is enabled.

B) view the information loaded by the kernel module

Command: lsmod | grep ipv6

2. Disable ipv6

Add at the end of/etc/modprobe. d/dist. conf

  1. alias net-pf-10 off
  2. alias ipv6 off 

You can use the vi editor or the following command:

Cat <EOF>/etc/modprobe. d/dist. conf

Alias net-pf-10 off

Alias ipv6 off

EOF

Disable Firewall

  1. chkconfig iptables stop
  2. chkconfig iptables off

Restart the server after modification:

  1. reboot
Hadoop02-hadoop06 needs to be set 4, install JDK to extract the compressed package to the specified directory, and then edit the environment variable
  1. vi /etc/proflie
Append the environment variable to the end of the file
  1. export JAVA_HOME=/soft/jdk1.7.0_80/
  2. export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
Refresh
  1. source /etc/profile
You can use java-version to test whether the installation is correct.

For more details, please continue to read the highlights on the next page:

  • 1
  • 2
  • Next Page

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.