Hadoop + Zookeeper)

Hadoop + Zookeeper)

I. Overview

1.1 single point of failure in Hadoop1.0

NameNode in Hadoop is like a human heart. It is very important and never stops working. In the hadoop1 era, there is only one NameNode. If the NameNode data is lost or cannot work, the whole cluster cannot be recovered. This is a single point of failure in hadoop1 and also an unreliable performance in hadoop1. Shows the architecture of hadoop1.0;

1.2 solution of hadoop2.0 to Single Point Problem of hadoop1.0

To solve the single point of failure (spof) Problem in hadoop1, the new NameNode in hadoop2 is no longer only one, but can have multiple NameNode (currently only two are supported ). Each has the same function. One is active and the other is standby. When the cluster is running, only the active NameNode works normally, and the standby NameNode Is In standby state, and the data of the active NameNode is synchronized at all times. Once the active NameNode cannot work, the standby state NameNode can be changed to active state through manual or automatic failover, and the work can continue. This is highly reliable.

1.3 Use JournalNode to share NameNode (Active and Standby) data

In Hadoop2.0, two NameNode data are actually shared in real time. The new HDFS adopts a sharing mechanism. The Quorum Journal Node (JournalNode) cluster or Nnetwork File System (NFS) is shared. NFS is at the operating system level, while JournalNode is at the hadoop level. Here we use the JournalNode cluster for data sharing (this is also the mainstream practice ). Shows the architecture of JournalNode.

For data synchronization, two NameNode communicate with each other through an independent process called JournalNodes. When the namespace of the active NameNode has any modification, most JournalNodes processes are notified. The NameNode in standby status can read the change information in JNs, constantly monitor the changes in the edit log, and apply the changes to your namespace. Standby ensures that the namespace status is fully synchronized when a cluster error occurs.

1.4 failovers between NameNode

For HA clusters, it is critical to ensure that only one NameNode is active at the same time. Otherwise, the data status of the two NameNode may be different, and data may be lost or incorrect results may be generated. To ensure this, ZooKeeper must be used. First, both NameNode in the hdfs cluster are registered in ZooKeeper. When the active NameNode fails, ZooKeeper can detect this situation, it automatically switches the NameNode In the standby state to the active state.

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Ii. Build a Hadoop (HA) Cluster

2.1 configuration details

Host Name IP NameNode DataNode Year Zookeeper JournalNode

Mast1	192.168.177.131	Yes	Yes	No	Yes	Yes
Mast2	192.168.177.132	Yes	Yes	No	Yes	Yes
Mast3	192.168.177.20.	No	Yes	Yes	Yes	Yes

2.2 install jdk

(Omitted) install jdk and configure Environment Variables

2.2 SSH Login-free

(Omitted), refer:

2.4 Zookeeper cluster Construction

(Omitted), for reference, this is my solr cluster deployment. It also uses zookeeper for management. zookeeper has the same steps as the operation here. Finally, my zoo. cfg file is as follows:

[Hadoop @ Mast1 conf] $ cat zoo. cfg
# The number of milliseconds of each tick
TickTime = 2000
# The number of ticks that the initial
# Synchronization phase can take
InitLimit = 10
# The number of ticks that can pass
# Sending a request and getting an acknowledgement
SyncLimit = 5
# The directory where the snapshot is stored.
DataDir =/home/hadoop/zookeeper/data
DataLogDir =/home/hadoop/zookeeper/datalog
# The port at which the clients will connect
ClientPort = 2181
Server.1 = mast1: 2888: 3888
Server. 2 = mast2: 2888: 3888
Server.3 = mast3: 2888: 3888

2.5 configure the Hadoop configuration file
First configure the mast1 machine. Then, copy the configuration environment to mast2 and mast3!
The configuration of hadoop2.0 is stored in ~ Under the/etc/hadoop directory,

Core. xml

<Configuration>
<! -- Specify the hdfs nameservice as ns -->
<Property>
<Name> fs. defaultFS </name>
<Value> hdfs: // ns </value>
</Property>
<! -- Specify the temporary directory for storing hadoop data -->
<Property>
<Name> hadoop. tmp. dir </name>
<Value>/home/hadoop/workspace/hdfs/temp </value>
</Property>

<Property>
<Name> io. file. buffer. size </name>
<Value> 4096 </value>
</Property>
<! -- Specify the zookeeper address -->
<Property>
<Name> ha. zookeeper. quorum </name>
<Value> mast1: 2181, mast2: 2181, mast3: 2181 </value>
</Property>
</Configuration>

Hdfs-site.xml

<Configuration>
<! -- Specify the nameservice of hdfs as ns, which must be consistent with that in the core-site.xml -->
<Property>
<Name> dfs. nameservices </name>
<Value> ns </value>
</Property>
<! -- Ns has two NameNode, nn1 and nn2 -->
<Property>
<Name> dfs. ha. namenodes. ns </name>
<Value> nn1, nn2 </value>
</Property>
<! -- RPC communication address of nn1 -->
<Property>
<Name> dfs. namenode. rpc-address.ns.nn1 </name>
<Value> MAID: 9000 </value>
</Property>
<! -- Nn1 http Communication address -->
<Property>
<Name> dfs. namenode. http-address.ns.nn1 </name>
<Value> MAID: 50070 </value>
</Property>
<! -- RPC communication address of nn2 -->
<Property>
<Name> dfs. namenode. rpc-address.ns.nn2 </name>
<Value> MAID: 9000 </value>
</Property>
<! -- Nn2 http Communication address -->
<Property>
<Name> dfs. namenode. http-address.ns.nn2 </name>
<Value> MAID: 50070 </value>
</Property>
<! -- Specify the storage location of NameNode metadata on JournalNode -->
<Property>
<Name> dfs. namenode. shared. edits. dir </name>
<Value> qjournal: // mast1: 8485; mast2: 8485; mast3: 8485/ns </value>
</Property>
<! -- Specify the location where JournalNode stores data on the local disk -->
<Property>
<Name> dfs. journalnode. edits. dir </name>
<Value>/home/hadoop/workspace/journal </value>
</Property>
<! -- Enable automatic failover when NameNode fails -->
<Property>
<Name> dfs. ha. automatic-failover.enabled </name>
<Value> true </value>
</Property>
<! -- Implementation of Automatic Switch upon configuration failure -->
<Property>
<Name> dfs. client. failover. proxy. provider. ns </name>
<Value> org. apache. hadoop. hdfs. server. namenode. ha. ConfiguredFailoverProxyProvider </value>
</Property>
<! -- Configure the isolation mechanism -->
<Property>
<Name> dfs. ha. fencing. methods </name>
<Value> sshfence </value>
</Property>
<! -- Ssh Login-free is required to use the isolation mechanism -->
<Property>
<Name> dfs. ha. fencing. ssh. private-key-files </name>
<Value>/home/hadoop/. ssh/id_rsa </value>
</Property>

<Property>
<Name> dfs. namenode. name. dir </name>
<Value> file: // home/hadoop/workspace/hdfs/name </value>
</Property>

<Property>
<Name> dfs. datanode. data. dir </name>
<Value> file: // home/hadoop/workspace/hdfs/data </value>
</Property>

<Property>
<Name> dfs. replication </name>
<Value> 2 </value>
</Property>
<! -- Enable the WebHDFS (rest api) function on NN and DN, not required -->
<Property>
<Name> dfs. webhdfs. enabled </name>
<Value> true </value>
</Property>
</Configuration>

Mapred-site.xml

<Configuration>
<Property>
<Name> mapreduce. framework. name </name>
<Value> yarn </value>
</Property>
</Configuration>

Yarn-site.xml

<Configuration>
<! -- Specify the method for loading the server when nodemanager is started as shuffle server -->
<Property>
<Name> yarn. nodemanager. aux-services </name>
<Value> mapreduce_shuffle </value>
</Property>
<! -- Specify the resourcemanager address -->
<Property>
<Name> yarn. resourcemanager. hostname </name>
<Value> mast3 </value>
</Property>
</Configuration>

Slaves

[Hadoop @ Mast1 hadoop] $ cat slaves
Mast1
Mast2
Mast3

Modify JAVA_HOME

Add JAVA_HOME configuration in file hadoop-env.sh and yarn-env.sh respectively

# Export JAVA_HOME =$ {JAVA_HOME} -- original
Export JAVA_HOME =/usr/local/java/jdk1.7.0 _ 67

Although the $ {JAVA_HOME} environment variable is configured by default, when hadoop is started, it will prompt that it cannot be found. There is no way to specify an absolute path, which is required.

Configure the hadoop environment variables. For more information, see my configuration.

[Hadoop @ Mast1 hadoop] $ vim ~ /. Bash_profile
Export HADOOP_HOME = "/home/hadoop/hadoop-2.5.2"
Export PATH = $ HADOOP_HOME/bin: $ HADOOP_HOME/sbin: $ PATH clear link
Export HADOOP_COMMON_LIB_NATIVE_DIR = $ HADOOP_HOME/lib/native
Export HADOOP_OPTS = "-Djava. library. path = $ HADOOP_HOME/lib"

Copy configurations to mast2 and mast3

Scp-r ~ /. Bash_profile hadoop @ mast2:/home/hadoop/
Scp-r ~ /. Bash_profile hadoop @ mast3:/home/hadoop/
Scp-r $ HADOOP_HOME/etc/hadoop @ mast2:/home/hadoop/hadoop-2.5.2/etc/
Scp-r $ HADOOP_HOME/etc/hadoop @ mast3:/home/hadoop/hadoop-2.5.2/etc/

Now that Hadoop is configured, the cluster is started.

For more details, please continue to read the highlights on the next page:

1 2 Next Page