Cainiao cloud computing 18th: Hadoop2.5.0HA cluster installation Chapter 4

Cainiao cloud computing 18th: Hadoop2.5.0HA cluster installation Chapter 4
Cainiao cloud computing 18th: Hadoop 2.5.0 HA cluster installation Chapter 2

Cheungmine

0 Introduction

Installing a Hadoop high-availability cluster in a production environment has always been a meticulous task that requires extreme patience and effort. Although there are many documents to teach you how to do this step by step, there are also a few hundred secrets. There are no ready-made tools, but for anyone I like to know the details, I can use other people's things to write well and not write well, and there is a problem, it is difficult to find errors. Manual installation of Hadoop clusters requires some experience in Linux. For those who are not familiar with Linux at all, it must be daunting. Therefore, this article describes how to install and deploy the latest Hadoop cluster in real scenarios. It is also a full record of my Hadoop automated installation script process. I do not like to repeat the manufacturing wheel. In most cases, I use ready-made items. However, when I see that Hadoop clusters and server-side development require a lot of manpower to deploy and maintain various software, I feel that this is supposed to free up more labor.

First of all, the HA cluster environment in this article is RHEL6.4 Virtual Machine (KVM). The configurations of each virtual machine are identical, except that the IP address, MAC address, and HOSTNAME are different. The vm host is Ubuntu14.04 Desktop on my laptop. My notebook has 8 CPU cores and 16 GB memory. The method in this article should also be applicable to real machines (RHEL6.4, 6.5.

1 Preparation 1.1 Cluster Planning

First, prepare the RHEL6.4 server. Because it is oriented to the production environment, at least five RHEL6.4 are required, and at least three are required for testing purposes only. I will demonstrate the configuration of five servers here. For more information about how to create a VM, see my series of articles:

Cainiao cloud computing 15th: create and manage Redhat virtual machines on Ubuntu

Cainiao cloud computing 17th: RHEL clones the VM and changes the NIC address

Cainiao cloud computing 8: KVM Virtual Machine of Ubuntu Server12.10

First, you need some preparation knowledge.

The node that stores data in Hadoop is called DataNode (DN), and the node that stores metadata is called NameNode (NN ). The metadata in NameNode consists of two parts: File Information and Block information. The file directory is the same as the traditional file directory, including the file name, Creation Time, and ACL information. It also contains all blockids of the file. BlockId ing maintains the complex factors of blockId, the current number of backups, and the data nodes of these backups. The first part of the information is generated when the client performs operations and is permanently stored in the local disk. Later, some information is stored only in the memory, and is generated by the datanode to report its block information to the heartbeat between namenode. There are many DataNode, and generally only one NameNode. To prevent NameNode from single point of failure and ensure high availability of HDFS, You need to synchronize and back up the previous part of the data. The standby role is secondnamenode. Under normal circumstances, the master will write disk operation logs and maintain the corresponding structure in the memory. When the server goes down, the memory image can be completely restored through log replay. However, if the log is very large and the recovery time is relatively long, the progress of the entire failover will be affected. Therefore, the memory image needs to be flushed to the disk after a period of time. When the image is restored next time, it is directly loaded into the memory, and then the latest log is replayed. The simplest memory image is to lock the memory and write it to the disk one by one. However, this may take a long time (especially when the memory image is large) and cannot respond to the request. Another way is to copy the memory image without locking it. The original memory will continue to be read and written, and the new memory will be gradually written to the disk. This will not impede Client requests, but will consume twice the memory usage. In fact, because second namenode and masternamenode have the same memory image in the Hot Standby state, this checkpoint operation is performed in second namenode. When the second namenode completes the checkpoint job and generates a new fsimage, it will request masternamenode to update the fsimage and delete the old edit log. There are currently two high availability mechanisms: the first is to use NFS. When the master namenode writes its own metadata log, it also writes the log to nfs. In this way, when the master namenode goes down, the second namenode can also find metadata from nfs and provide services. This method has the advantages of being simple. As a service, nfs only has a single point of writing. The disadvantage is that the namenode nfs device has high requirements and cannot be configured with the same model as the cluster, additionally, You Need To Configure nfs. The second is QJM. To put it simply, when the master namenode writes logs, it also writes data to other namenode disks. The second method does not have the disadvantages of the first solution for heterogeneous devices, but requires a complex mechanism to ensure data consistency. Based on the characteristics of HDFS, this mechanism implements the simple Paxos protocol to ensure the consistency of distributed logs. There are two roles in the design: 1. the node where the JournalNode actually writes logs is responsible for storing logs to the underlying disk, which is equivalent to the acceptor in the paxos protocol. 2. QuorumJournalManager runs in NameNode. It is responsible for sending log write requests to all journalnodes and executing the write Fencing and log synchronization functions, which is equivalent to proposer in paxos. Our job is to build on the second mechanism QJM (Quorum Journal Manager.

Create a configuration file hacl. cfg to describe all available server nodes and the software to be deployed:

################################################################################# hacl.cfg# settings for hadoop cluster (hadoop 2.5.0).# based on work of cdh5.2:# -- http://www.cloudera.com/content/cloudera/en/downloads/cdh/cdh-5-2-0.html# copyright 2014-, cheungmine@hgdb.net, all rights reserved.################################################################################[hacl-global]username=rootpassword=abc123nodes=5[hdfs-namenode]refer=1[hdfs-secondarynamenode]refer=2[hdfs-journalnode]refer=1,2,3[zookeeper]refer=1,2,3[yarn-resourcemanager]refer=1,2,3[yarn-nodemanager]refer=1,2,3# hacl nodes[node:1]ipv4addr=192.168.122.101hostname=hacl-master.hgdb.net[node:2]ipv4addr=192.168.122.102hostname=hacl-secondary.hgdb.net[node:3]ipv4addr=192.168.122.103hostname=hacl-datanode1.hgdb.net[node:4]ipv4addr=192.168.122.104hostname=hacl-datanode2.hgdb.net[node:5]ipv4addr=192.168.122.105hostname=hacl-datanode3.hgdb.net

The preceding configuration file defines the role of each node in the HACL as follows:
IPhostname NameNodeJournal NodeDataNodeResource ManagerNodeManagerZooKeeper192.168.122.101JN1 ZK1 NN1Active
√ 192.168.122.102JN2 ZK2 NN2StandbyNN √ 192.168.122.103JN3 ZK3 √ Note: DataNode and NodeManager share one machine in the production environment, HA's NN1 and NN2 machines, JN [1-3] Each occupies one machine (or the other two and NN share), and ZK [1-3] Each occupies one machine. Conjecture: the sharing of NN and JN on a single machine may result in a slightly higher performance. 1.2 software download: Download all software locally and configure it as the yum source. I am using cdh5.2 provided by cloudera, and ,:

Http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html

I downloaded all the RPM packages to the local device. Run the following command ($ {project-dir}/bin/cdh5.2-downloads. sh ):

#!/bin/bash## @file#   cdh5.2-downloads.sh## @date#   2014-10-26## @author#   cheungmine@hgdb.net## @version#   0.0.1pre## download all RPMS from:# -- http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.2.0/RPMS/x86_64/## TODO:#   should parse above page and to download packages automatically.#################################################################################CDH5_2_URL_PREFIX="http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.2.0/RPMS/x86_64/"DOWNLOADS_PREFIX="../downloads/cdh5.2-RPMS-x86_64"# download function#   $1 - package namefunction dl() {    # get dir of this shell    bin_dir=$(cd "$(dirname "$0")"; pwd)    # save packages to dir    pkgs_dir=$bin_dir"/"$DOWNLOADS_PREFIX    echo $pkgs_dir    # only download new package (wget -N)    echo "++++ downloading package: "$1    wget -N $CDH5_2_URL_PREFIX$1 -P $pkgs_dir}dl_pkgs="bigtop-jsvc-0.6.0+cdh5.2.0+572-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        bigtop-jsvc-debuginfo-0.6.0+cdh5.2.0+572-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hadoop-0.20-conf-pseudo-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-0.20-mapreduce-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-0.20-mapreduce-jobtracker-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-0.20-mapreduce-jobtrackerha-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-0.20-mapreduce-tasktracker-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-0.20-mapreduce-zkfc-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-client-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-conf-pseudo-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-debuginfo-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-doc-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-datanode-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-fuse-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-journalnode-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-namenode-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-nfs3-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-secondarynamenode-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-hdfs-zkfc-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-httpfs-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-kms-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-kms-server-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-libhdfs-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-libhdfs-devel-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-mapreduce-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-mapreduce-historyserver-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-yarn-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-yarn-nodemanager-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-yarn-proxyserver-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hadoop-yarn-resourcemanager-2.5.0+cdh5.2.0+551-1.cdh5.2.0.p0.46.el6.x86_64.rpm/        hbase-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hbase-doc-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hbase-master-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hbase-regionserver-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hbase-rest-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hbase-thrift-0.98.6+cdh5.2.0+55-1.cdh5.2.0.p0.33.el6.x86_64.rpm/        hue-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-beeswax-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-common-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-doc-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-hbase-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-impala-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-pig-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-plugins-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-rdbms-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-search-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-security-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-server-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-spark-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-sqoop-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        hue-zookeeper-3.6.0+cdh5.2.0+509-1.cdh5.2.0.p0.37.el6.x86_64.rpm/        impala-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-catalog-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-debuginfo-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-server-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-shell-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-state-store-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        impala-udf-devel-2.0.0+cdh5.2.0+0-1.cdh5.2.0.p0.49.el6.x86_64.rpm/        zookeeper-3.4.5+cdh5.2.0+82-1.cdh5.2.0.p0.40.el6.x86_64.rpm/        zookeeper-debuginfo-3.4.5+cdh5.2.0+82-1.cdh5.2.0.p0.40.el6.x86_64.rpm/        zookeeper-native-3.4.5+cdh5.2.0+82-1.cdh5.2.0.p0.40.el6.x86_64.rpm/        zookeeper-server-3.4.5+cdh5.2.0+82-1.cdh5.2.0.p0.40.el6.x86_64.rpm"dl_pkgs_array=($(echo $dl_pkgs | tr '/' ' ' | tr -s ' '))dl_pkgs_num=${#dl_pkgs_array[@]}########################################################################echo "start downloading $dl_pkgs_num packages..."dl_num=0for ((i=0; i<$dl_pkgs_num; i++))do    pkg=${dl_pkgs_array[$i]}    echo $pkg    dl $pkg    ((dl_num++))doneecho "$dl_num packages downloaded successfully."

Conclusion 1.3

So far, the five node computers RHEL6.4 have been prepared (correct IP addresses and network connectivity), and the required software has been downloaded (cdh5.2 ). The apache version of hadoop is not used. Because the apache version needs to be compiled from the source code to a 64-bit version, all the efforts are taken as the principle. Therefore, the RMP package prepared by cdh is used.

Next, we officially enter the deployment stage. In the following chapter, we will see:

2 deploy Hadoop HA cluster 2.1 node computer preprocessing 2.2 ssh password-free login 2.3 install jdk