As you know, Namenode has a single point of failure in the Hadoop system, which has been a weakness for high-availability Hadoop. This article discusses several solution that exist to solve this problem. 1. Secondary NameNode principle: secondary NN periodically reads the editlog from the NN, merging with the image that it stores to form a new metadata image advantage: The earlier version of Hadoop comes with its own, simple configuration, Basic no additional resources (can be shared with Datanode machine) Disadvantage: Slow recovery time, there will be some data loss 2. Backup NameNode principle: Backup NN real-time get editlog, when nn down, manually switch to backup NN; Pros: from hadoop0.21 to provide this scenario, there will be no loss of data Cons: Because the location information of the block needs to be obtained from the datanode, it is slower (depending on the amount of data) 3 when switching to backup NN. Avatar NameNode principle: This is an HA scheme provided by Facebook, which puts client access to Hadoop Editlog in NFS, Standby nn can get editlog;datanode in real time and need to work with active NN and standby NN report block information; Advantages: Information is not lost, recovery fast (seconds) Disadvantage: Facebook based on Hadoop0.2 development, the deployment of a little trouble; additional machine resources are required, and NFS becomes another single point (but with a low failure rate) of 4. Hadoop2.0 directly supports standby NN, draws on Facebook's avatar, and then makes some improvements: information is not lost, recovery is fast (seconds), simple deployment
--------------------------------------------------------------------------------------------------------------- -----------------------------------------------------
Learn more about one solution to Hadoop namenode single point problem Avatarnode
Requirements: Implement Namenode metadata backup, solve the problem that namenode single-point outage causes the cluster not to be available. Scenario Description: When the Namenode server goes down, we can quickly refactor the new Namenode using the Namenode backup metadata. 1. Hadoop itself provides a solution that can utilize Secondarynamenode's backup data to recover Namenode metadata, but because checkpoint (at each Checkpoint Secondarynamenode Merge and synchronize Namenode data), Secondarynamenode backup data is not always synchronized with Namenode, This means that Secondarynamenode may lose data for a period of time when the Namenode is down, depending on the checkpoint cycle. We can reduce the checkpoint cycle to decrease the amount of data lost, but because of the performance of each checkpoint, and this solution can not fundamentally solve the problem of data loss. So if the demand does not allow the loss of such data, such a scheme can be directly disregarded. 2. Another solution offered by Hadoop is NFS, a scheme for instantly backing up namenode metadata, setting up multiple data directories (including NFS directories), and allowing Namenode to write multiple directories at the same time when persisting metadata. The advantage of this scheme over the first scenario is that it avoids the loss of data (we will not discuss the possibility of NFS itself losing data for the time being, after all, this is a very small probability). Since it is possible to solve the problem of data loss, it is explained that this scheme is feasible in principle. Download source code
Https://github.com/facebook/hadoop-20
Deployment environment
Machine 4 Units
hadoop1-192.168.64.41 Avatarnode (primary)
hadoop2-192.168.64.42 Avatadatanode
hadoop3-192.168.64.43 Avatadatanode
hadoop4-192.168.64.67 Avatarnode (Standby)
Related Resources and description
The following is a brief introduction to the Avatar Scenario deployment.
1. First about the avatar scenario the backup of Hadoop is a single point of backup to DFS and does not include mapred, because Hadoop itself does not have a mechanism for handling Jobtracker single points of failure.
The 2.AvatarNode inherits from the Namenode, not the Namenode modification, avatardatanode the same. Therefore, the startup mechanism of Avatar is independent of the startup mechanism of Hadoop itself.
3. In the avatar scenario, Secondarynamenode's responsibilities are included in the standby node, so there is no need to start a secondarynamenode independently.
4.AvatarNode must have NFS support for sharing the transaction log (Editlog) between two nodes.
5.FB provided by the Avatar source code temporarily does not realize the automatic switching between primary and standby, you can use the Zookeeper lease mechanism to achieve automatic switching.
Switching between 6.Primary and standby only includes switching from standby to primary, and does not support switching from primary state to standby state.
7.AvatarDataNode does not use VIP and avatarnode communication, but directly with the primary and standby communication, it is necessary to use the VIP drift scheme to block the transition between two nodes in the process of IP transformation problem. With regard to the integration with zookeeper, officials say it will be released in a later version.
For a more detailed introduction to Avatarnode, please refer to http://blog.csdn.net/rzhzhz/article/details/7235789,
Third, compile
1. First modify the Hadoop root directory under build. XML, comment out 996 lines and 1000 lines. As follows:
<targetname= "Forrest.check" unless= "Forrest.home" depends= "Java5.check" > <!--fail message= "' Forrest.home ' is not defined. Pleasepass-dforrest.home=<base of Apache Forrest installation> to Antonthe command-line. "/--> </target> <target name= "Java5.check" unless= "Java5.home" > <!--fail message= "' java5.home ' is not defined. Forrest requires Java 5. Pass-djava5.home=<base of Java5 distribution> to Ant onthe command-line. "/--> </target>
2. In the root directory, enter the Ant jar (for compiling the package can refer to Build.xml code) to compile Hadoop, the compiled jar will be in the build directory (Hadoop-0.20.3-dev-core.jar), Copy the jar package to the Hadoop root and replace it with the original Jar (verbose, Hadoop starts by loading the class in the build directory, so when you modify the jar package by replacing the class, you remove the build directory first).
3. Enter the Src/contrib/highavailability directory to compile the avatar, and the compiled jar package will be in the Build/contrib/highavailability directory (hadoop-${version}- Highavailability.jar), copy the jar package to the Lib directory.
4. Distribute the jar packages compiled in 2, 3 steps to the appropriate directory for all machines in the cluster.
Iv. Configuration
1. Configure Hdfs-site.xml
<?xmlversion= "1.0"?>
<? Xml-stylesheettype= "text/xsl" href= "configuration.xsl"?>
<!--Put Site-specificproperty overrides in Thisfile. -
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/data/hadoop/hdfs/name</value>
<description>determineswhereon The local filesystem the DFS name node shouldstore the name table. Ifthis is a comma-delimited list ofdirectories and the name Tableis replicated in all of thedirectories, for redundancy
</description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/hadoop/facebook_hadoop_data/hdfs/data</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:50011</value>
<description> defaults to 50010, which is the Datanode listening port </description>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:50076</value>
<description> default is 50075, HTTP server port for Datanode </description>
</property>
<property>
<name>dfs.datanode.ipc.address</name>
<value>0.0.0.0:50021</value>
<description> default is 50020, IPC server port for Datanode </description>
</property>
<property>
<name>dfs.http.address0</name>
<value>192.168.64.41:50070</value>
</property>
<property>
<name>dfs.http.address1</name>
<value>192.168.64.67:50070 </value>
</property>
<property>
<name>dfs.name.dir.shared0</name>
<value>/data/hadoop/share/shared0</value>
</property>
<property>
<name>dfs.name.dir.shared1</name>
<value>/data/hadoop/share/shared1</value>
</property>
<property>
<name>dfs.name.edits.dir.shared0</name>
<value>/data/hadoop/share/shared0</value>
</property>
<property>
<name>dfs.name.edits.dir.shared1</name>
<value>/data/hadoop/share/shared1</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>
Defaultblock replication. The actual number of Replicationscan bespecified when the file is created. The default isused Ifreplicationis not specified in Create time
</description>
</property>
</configuration>
Parameter description:
1) dfs.name.dir.shared0
Avatarnode (Primary) Metadata Store directory, note cannot be the same as Dfs.name.dir directory
2) dfs.name.dir.shared1
Avatarnode (Standby) Metadata Store directory, note cannot be the same as Dfs.name.dir directory
3) dfs.name.edits.dir.shared0
Avatarnode (Primary) edits file storage directory, default to Dfs.name.dir.shared0
4) dfs.name.edits.dir.shared1
Avatarnode (Standby) edits file storage directory, default to Dfs.name.dir.shared1
5) Dfs.http.address0
Avatarnode (Primary) HTTP monitoring address
6) Dfs.http.address1
Avatarnode (Standby) HTTP monitoring address
7) Dfs.namenode.dn-address0/dfs.namenode.dn-address1
Although it is involved in the Avatar source code, it has not been used for the time being
2. Configure Core-site.xml
<?xmlversion= "1.0"?>
<? Xml-stylesheettype= "text/xsl" href= "configuration.xsl"?>
<!--Putsite-specificproperty overrides in the this file. -
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>a baseforother temporary directories.
</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.64.41:9600</value>
<description>the name Ofthedefault file system. eitherthe literal string "local" or a host:port for DFS.
</description>
</property>
<property>
<name>fs.default.name0</name>
<value>hdfs://192.168.64.41:9600</value>
<description>the name Ofthedefault file system. eitherthe literal string "local" or a host:port for DFS.
</description>
</property>
<property>
<name>fs.default.name1</name>
<value>hdfs://192.168.64.67:9600</value>
<description>the name Ofthedefault file system. eitherthe literal string "local" or a host:port for DFS.
</description>
</property>
</configuration>
Parameter description:
1) fs.default.name
The current Avatarnode IP address and port number, that is, primary and standby are configured for their respective IP addresses and port numbers.
2) FS.DEFAULT.NAME0
Avatarnode (Primary) IP address and port number
3) fs.default.name1
Avatarnode (Standby) IP address and port number
3. Because it does not involve mapred, so mapred-site. XML does not need to be modified for the original cluster configuration.
4. Distribute the modified configuration file to the cluster node and set up the appropriate directory in the configuration file on the primary and standby nodes.
5. Build NFS to achieve data sharing between the primary and standby shared0 directories. For NFS Configuration please refer to http://blog.csdn.net/rzhzhz/article/details/7056732
6. Format primary and standby, where you can use the Format command of Hadoop itself, or you can use the Avatarnode Format command (Bin/hadooporg.apache.hadoop.hdfs.avatarshell- format), but at this point the shared1 directory cannot be empty, which is a bit redundant here. It is recommended that the format command for Hadoop itself be formatted on primary and that the files under the name directory be copied to the Shared0 directory. Then copy the files under the Shared0 directory to the shared1 directory on the standby.
Five, start
1. Since the single point of Jobtracker is not involved, we only start the HDFS related thread here. Primary,standby two Namenode (here standby includes Secondarynamenode's responsibilities) and 3 Avatardatanode data nodes.
2. Start Avatarnode (Primary) under the Primary node Hadoop root directory
Bin/hadooporg.apache.hadoop.hdfs.server.namenode.avatarnode–zero
3. Start Avatarnode (Standby) under the Standby node Hadoop root directory
Bin/hadooporg.apache.hadoop.hdfs.server.namenode.avatarnode-one–standby
4. Start Avatardatanode in the Hadoop root directory of the data node in turn
Bin/hadooporg.apache.hadoop.hdfs.server.datanode.avatardatanode
5. Other Related commands
Bin/hadoop Org.apache.hadoop.hdfs.server.namenode.AvatarNode, the following optional parameters are
[-standby] | [-sync] | [-zero] | [-one] | [-format] | [-upgrade] | [-rollback] | [-finalize] | [-importcheckpoint]
# #查看当前AvatarNode的状态
1) Bin/hadoop Org.apache.hadoop.hdfs.avatarshell–showavatar
# #primary Upgrade the current standby node to the primary node
2) Bin/hadooporg.apache.hadoop.hdfs.avatarshell-setavatar
3) Bin/hadooporg.apache.hadoop.hdfs.avatarshell-setavatar Standby
Cluster testing
1. Accessing a clustered Web page
(Primary) http://hadoop1-virtual-machine:50070
(Standby) http://hadoop5-virtual-machine:50070
It is visible that all Avatardatanode are registered to two namenode,primary in a normal state, while standby is in the SafeMode state and can only be read and not writable. The status of the current Avatarnode (primary or standby) can be viewed through the Avatarshell command.
2. Store the relevant data to the cluster, the cluster is working properly.
3. kill off the primary node Avatartnode thread, in standby the current upgrade to Prirmary, the data is not lost, the cluster is working properly (at this time the Web is not able to access the file system, You can view the cluster data through the shell command). However, since the Avatar has conversion restrictions, can only be converted from standby to primary, so after a fault, the standby up to primary node and can not be re-downgraded to standby, so can not achieve the kind of free switching like master/slave.
