Hadoop 2.7.1 high-availability installation and deployment

Hadoop 2.7.1 high-availability installation and deployment
Hadoop cluster planning goals: create two NameNode for high availability. One NameNode fails and the other can be started. One runs Yarn, three DataNode, and three Zookeeper clusters for high availability. Hadoop2 usually consists of two NameNode, one in active state and the other in standby state. Active NameNode provides external services, while Standby NameNode does not provide external services. It only synchronizes the status of active namenode, so that it can quickly switch when it fails. A zookeeper cluster is also configured for ZKFC (DFSZKFailoverController) failover. When Active NameNode fails, the Standby NameNode status is automatically switched to standby status.
Install hadoop and Zookeeper in/soft;

 

Http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

0. Install the CentOS6.5 and 64-bit operating systems in the system environment with minimal installation. To achieve the goal, use the VMware Virtual Machine to build 6 servers. Therefore, we recommend that the servers have at least 8 GB of memory; vmware uses VMware.®Workstation 11.0.0 build-2305329; the network configuration is as follows:
The virtual machine configuration is as follows: 1. Synchronize the Machine Time

1. Yum install-y ntp # install Time Service
2. Ntpdate us.pool.ntp.org # synchronization time

2. Set the Host IP Address

1. vi /etc/sysconfig/network-scripts/ifcfg-eth0

1. DEVICE=eth0
2. IPADDR=192.168.8.101
3. NETMASK=255.255.255.0
4. GATEWAY=192.168.8.2
5. HWADDR=00:0C:29:56:63:A1
6. TYPE=Ethernet
7. UUID=ecb7f947-8a93-488c-a118-ffb011421cac
8. ONBOOT=yes
9. NM_CONTROLLED=yes
10. BOOTPROTO=none

Then restart the network service.

1. service network restart

View ip configuration

1. ifconfig eth0

1. eth0 Link encap:EthernetHWaddr00:0C:29:6C:20:2B
2. inet addr:192.168.8.101Bcast:192.168.8.255Mask:255.255.255.0
3. inet6 addr: fe80::20c:29ff:fe6c:202b/64Scope:Link
4. UP BROADCAST RUNNING MULTICAST MTU:1500Metric:1
5. RX packets:777 errors:0 dropped:0 overruns:0 frame:0
6. TX packets:316 errors:0 dropped:0 overruns:0 carrier:0
7. collisions:0 txqueuelen:1000
8. RX bytes:70611(68.9KiB) TX bytes:49955(48.7KiB)

This indicates that the IP address we configured is successful. Note: five virtual machines are cloned using vmware, and an error occurs when configuring the NIC: "Bring up interface eth0: device eth0 does not seem to be present, delaying initialization "solution: Step 1 delete the file 70-persistent-net.rules

1. rm -f /etc/udev/rules.d/70-persistent-net.rule

Step 2 modify ifcfg-eth0

1. vim /etc/sysconfig/network-scripts/ifcfg-eth0

Step 3: restart the server

1. reboot

In this way, you can set a new IP address. 3. Set the host name to hadoop01.

1. vi /etc/sysconfig/network


Modify the hostname:

1. NETWORKING=yes
2. HOSTNAME=hadoop01
3. NETWORKING_IPV6=no


To modify hosts

1. vi /etc/hosts

1. 127.0.0.1 localhost
2. 192.168.8.101 hadoop01 192.168.8.102 hadoop02 192.168.8.103 hadoop03 192.168.8.104 hadoop04 192.168.8.105 hadoop05 192.168.8.106 hadoop06

Disable ipv6

1. Check whether ipv6 is enabled in the system.

A) view the ENI attributes

Command: ifconfig

Note: "inet6 addr:..." is available :....... The ipv6 feature is enabled.

B) view the information loaded by the kernel module

Command: lsmod | grep ipv6

2. Disable ipv6

Add at the end of/etc/modprobe. d/dist. conf

1. alias net-pf-10 off
2. alias ipv6 off 

You can use the vi editor or the following command:

Cat <EOF>/etc/modprobe. d/dist. conf

Alias net-pf-10 off

Alias ipv6 off

EOF

Disable Firewall

1. chkconfig iptables stop
2. chkconfig iptables off

Restart the server after modification:

1. reboot

Hadoop02-hadoop06 needs to be set 4, install JDK to extract the compressed package to the specified directory, and then edit the environment variable

1. vi /etc/proflie

Append the environment variable to the end of the file

1. export JAVA_HOME=/soft/jdk1.7.0_80/
2. export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

Refresh

1. source /etc/profile

You can use java-version to test whether the installation is correct.

For more details, please continue to read the highlights on the next page:

1 2 Next Page