website Link Hadoop profile home: Http://hadoop.apache.org/docs/r2.5.2/HDFS hapdoop ha full distribution configuration: http://hadoop.apache.org/docs/ R2.5.2/hadoop-project-dist/hadoop-hdfs/hdfshighavailabilitywithqjm.html Summary steps: Preparation: 1. Configuring the Java environment variable can be done in/etc/profile or configure in/root/.bash_profile 2. Configure password-free logins especially namenode can be password-free to each other 3. Prepare Zookeepera) three zookeeper:hadoop1,hadoop2,hadoop3b) Edit zoo.cfg configuration file Modify Datadir=/opt/zookeeper//customizable file location Server.1=node1:2888:3888server.2=node2:2888:3888server.3=node3 : 2,888:3,888 Create a myID file in the DataDir directory, with a file content of three-to-one (three-to-three server. After number) Hadoop configuration file 1. Configuring Java_homeexport in Hadoop-env.sh java_home=/usr/java/jdk1.7.0_792. Configuring Slaves: Host Mapping example of Datanode: Node2 node3 node43. Configuring Hdfs-site.xml<Configuration> <!--Configure service Name - < Property> <name>Dfs.nameservices</name> <value>Xiaobing</value> </ Property> <!--Configure permission shutdown to make access available to other users - < Property> <name>Dfs.permissions.enabled</name> <value>False</value> </ Property> <!--give Namenode a name - < Property> <name>Dfs.ha.namenodes.xiaobing</name> <value>Nn1,nn2</value> </ Property> <!--Establish RPC Data communication protocol for the first Namenode - < Property> <name>Dfs.namenode.rpc-address.xiaobing.nn1</name> <value>node1:8020</value> </ Property> <!--Establish RPC Data communication protocol for the second Namenode - < Property> <name>Dfs.namenode.rpc-address.xiaobing.nn2</name> <value>node4:8020</value> </ Property> <!--establish an HTTP protocol for the first Namenode - < Property> <name>Dfs.namenode.http-address.xiaobing.nn1</name> <value>node1:50070</value> </ Property> <!--establishing an HTTP protocol for the second Namenode - < Property> <name>Dfs.namenode.http-address.xiaobing.nn2</name> <value>node4:50070</value> </ Property> <!--Configure the hostname port of the journalnode, with a random file name - < Property> <name>Dfs.namenode.shared.edits.dir</name> <value>Qjournal://node2:8485;node3:8485;node4:8485/abcdir</value> </ Property> <!--Configuring the failover mechanism - < Property> <name>Dfs.client.failover.proxy.provider.xiaobing</name> <value>Org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </ Property> < Property> <name>Dfs.ha.fencing.methods</name> <value>Sshfence</value> </ Property> <!--Configure the local private key so that the failover controller can be checked by SSH Namenode - < Property> <name>Dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_dsa</value> </ Property> <!--Set automatic takeover - < Property> <name>Dfs.ha.automatic-failover.enabled</name> <value>True</value> </ Property> </Configuration>4. Configure Core-site.xml<Configuration> <!--Configure access Service name - < Property> <name>Fs.defaultfs</name> <value>Hdfs://xiaobing</value> </ Property> <!--Configuring the Nomenode fsimage file generation file location - < Property> <name>Hadoop.tmp.dir</name> <value>/home/data</value> </ Property> <!--Configuring the Journalnode edits file storage path - < Property> <name>Dfs.journalnode.edits.dir</name> <value>/opt/jn/data</value> </ Property> <!--Configuring host name ports for zookeeper - < Property> <name>Ha.zookeeper.quorum</name> <value>node1:2181,node2:2181,node3:2181</value> </ Property></Configuration>5. Start all zookeeper zkserver.sh start6. Starts all journalnode:hadoop-daemon.sh start Journalnode7. Format on any of the Namenode: H DFS namenode-format8. Copy the metadata that was just formatted to another namenode a) start the newly formatted namenodeb) on any unformatted Namenode: HDFs Namenode- BOOTSTRAPSTANDBYC) Start the second namenode9. Initialize Zkfc:hdfs ZKFC on one of the Namenode- FormatZK10. Stopping the above node: stop-dfs.sh11. Full boot: start-dfs.sh
hadoop2.x HDFs fully distributed HA Setup