Hadoop2.2.0 manually switch HA environment setup

Ssh-copy-id-i hadoop5 Meaning:

The meaning of executing ssh-copy-id-i hadoop5 on node HADOOP4 is to append the contents of the public key id_rsa.pub on HADOOP4 to HADOOP5 's authorization file Authorized_keys.

One of the functions implemented by zookeeper in HADOOP2 is the automatic switching of ha.
Journalnode is the implementation of data synchronization between active and standby two data nodes.
Build a manually switched ha (more than HADOOP1 cluster build up a Journalnode)

Machine allocation:

Namenode:hadoop4,hadoop5
Datanode:hadoop4,hadoop5,hadoop6
JOURNALNODE:HADOOP4,HADOOP5,HADOOP6, just to achieve ha, and the cluster does not have any relationship, at least three machines, odd number of
Resourcemanager:hadoop4
Nodemanager:hadoop4,hadoop5,hadoop6

1.1 Configuration file: (hadoop-env.sh,core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml)

1.1.1 Hadoop-env.sh Modify Java_home

Export JAVA_HOME=/USR/LOCAL/JDK

1.1.2 Core-site.xml

< Property>    <name>Fs.defaultfs</name>    <value>Hdfs://cluster1</value></ Property>< Property>    <name>Hadoop.tmp.dir</name>    <value>/usr/local/hadoop/tmp</value></ Property>

1.1.3 Hdsf-site.xml

<!--number of replicas default value 3 -< Property>    <name>Dfs.replication</name>    <value>2</value></ Property><!--there are several clusters configured -< Property>    <name>Dfs.nameservices</name>    <value>Cluster1</value></ Property><!--Specifies which Namenode nodes the cluster cluster1 have, and here is the logical name, which is not duplicated -< Property>    <name>Dfs.ha.namenodes.cluster1</name>    <value>hadoop101,hadoop102</value></ Property><!--Specify the address of the HADOOP101RPC -< Property>    <name>Dfs.namenode.rpc-address.cluster1.hadoop101</name>    <value>hadoop4:9000</value></ Property><!--Specify the address of the Hadoop101http -< Property>    <name>Dfs.namenode.http-address.cluster1.hadoop101</name>    <value>hadoop4:50070</value></ Property><!--Specify the address of the HADOOP102RPC -< Property>    <name>dfs.namenode.rpc-address.cluster1.hadoop102</name>    <value>hadoop5:9000</value></ Property><!--Specify the address of the Hadoop102http -< Property>    <name>dfs.namenode.http-address.cluster1.hadoop102</name>    <value>hadoop5:50070</value></ Property><!--whether automatic failback is enabled, that is, if the namenode fails, automatically switches to another Namenode -< Property>    <name>Dfs.ha.automatic-failover.enabled.cluster1</name>    <value>False</value></ Property><!--use Journalnode cluster information when specifying the Cluster1 two Namenode shared edits file directory -< Property>    <name>Dfs.namenode.shared.edits.dir</name>    <value>Qjournal://hadoop4:8485;hadoop5:8485;hadoop6:8485/cluster1</value></ Property><!--Specifies the disk path where the Journalnode cluster stores data when it shares the Namenode directory -< Property>    <name>Dfs.journalnode.edits.dir</name>    <value>/usr/local/hadoop/tmp/journal</value></ Property><!--need namenode switch, use SSH to switch -< Property>    <name>Dfs.ha.fencing.methods</name>    <value>Sshfence</value></ Property><!--Using ssh for failover, the key storage location used -< Property>    <name>Dfs.ha.fencing.ssh.private-key-files</name>    <value>/root/.ssh/id_rsa</value></ Property><!--Specifies which implementation class is responsible for performing a failover when the cluster1 fails -< Property>    <name>Dfs.client.failover.proxy.provider.cluster1</name>    <value>Org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></ Property>

hdsf-site.xml Configuration

1.1.4 Yarn-site.xml

< Property>    <name>Yarn.resourcemanager.hostname</name>    <value>Hadoop4</value></ Property>< Property>    <name>Yarn.nodemanager.aux-services</name>    <value>Mapreduce_shuffle</value></ Property>

1.1.5 Mapred-site.xml

Rename Mapred-site.xml.template to Mapred-site.xml
MV Mapred-site.xml.template Mapred-site.xml

<!---<property>        <name >mapreduce.framework.name</name>        <  Value>yarn</value></Property  >

1.1.6 Slaves

Hadoop4hadoop5hadoop6

1.1.7 Copy the Hadoop folder on HADOOP4 to the HADOOP5, HADOOP6 node

Direct copy:    scp-rq Hadoop hadoop5:/usr/local/    R: Whether to force overwrite    Q: Quiet can also write a script:    vi bcp.sh    #!/bin/sh    For ((I=5;i<=6; i++));    do scp-rq/usr/local/$1 hadoop$i:/usr/local/$2    echo "copy did at hadoop$i"    done:    chmod u+x bcp.sh    bcp.sh Hadoop.

1.2 Starting the Journalnode cluster

Perform hadoop/sbin/hadoop-daemon.sh start Journalnode on HADOOP4, HADOOP5 and HADOOP6 respectively

1.3 Format Namenode, start Namenode

Format on HADOOP4: Hadoop/bin/hdfs Namenode-format
HADOOP4 execution hadoop/sbin/hadoop-daemon.sh Start Namenode
Execute Hadoop/bin/hdfs namenode-bootstrapstandby on HADOOP5
HADOOP5 execution hadoop/sbin/hadoop-daemon.sh Start Namenode
Convert the standby state on HADOOP4 to the active state
Hadoop/bin/hdfs haadmin-failover--forceactive hadoop101 hadoop102 forceactive: Turn One into active,forcefence: swap states
Ll-h: Viewing File size

1.4 Start Datanode

Execute hadoop/sbin/hadoop-daemons.sh start Datanode on HADOOP4
HADOOP5 format command, actually copy fsimage and edits two files on HADOOP4 to HADOOP5

1.5 Starting ResourceManager and NodeManager

Execute hadoop/sbin/start-yarn.sh start ResourceManager on HADOOP4
Compared to the original cluster, a HA,HA is implemented by our Journalnode cluster,

Hadoop2.2.0 manually switch HA environment setup