Hadoop Secondary Namenode role

Light literally understood, it is easy for some beginners to assume that: Secondarynamenode (SNN) is the Namenode (NN) hot standby process. Not really. SNN is an integral part of the HDFS architecture, but is often misunderstood for its real purpose by name, in fact its real purpose is to save the namenode of the HDFs metadata information and to reduce the time of namenode restart. There is still some work to be done in the Hadoop process to configure and use SNN correctly. The default configuration for Hadoop allows the SNN process to run on the Namenode machine by default, but in this case, if this machine goes wrong, downtime is a major disaster for recovering the HDFs file system, and the better way is to configure the SNN process to run on another machine.

In Hadoop, Namenode is responsible for persisting the metadata of HDFs and handling interactive feedback from the client for various operations on HDFS. To ensure the speed of the interaction, the metadata of the HDFs file system is load into the memory of the Namenode machine, and the in-memory data is saved to disk for persistent storage. To ensure that this persistence process does not become a bottleneck for HDFS operations, Hadoop does this by not persisting the snapshot of the current file system at any one time, The operation list is saved to a file called Editlog in Namenode in the near time. When restarting Namenode, in addition to the load fsimage, the HDFs operation recorded in this Editlog file is replay to restore the final state before the HDFs restart.

In Secondarynamenode, the operations of HDFs recorded in Editlog are periodically merged into one checkpoint and then emptied Editlog. So the Namenode reboot will load the latest checkpoint, and replay the HDFs operation recorded in Editlog, because the Editlog record is from the last checkpoint to the current list of operations, so it will be relatively small. Without this periodic merging of SNN, it would take a long time to restart Namenode each time. This periodic merger can reduce the time to restart. The integrity of the HDFS system can also be ensured.

This is what Secondarynamenode is doing. Therefore, SNN can not share the pressure on the interactive operation of HDFs on Namenode. However, when the Namenode machine goes down or the namenode process goes wrong, the Namenode daemon process can manually copy a SNN from metadata to recover the HDFs file system. as to why the SNN process is to be run on a non-namenode machine, this is mainly due to two considerations: Extensibility: Creating a new HDFs snapshot requires that the metadata information in the Namenode load to memory be copied all over again, so that the required memory needs to be the same as the memory that the Namenode occupies. Since the memory allocated to the Namenode process is actually a limitation on the HDFs file system, if the distributed file system is very large, the memory of the Namenode machine may be occupied by the Namenode process. Fault tolerance: When SNN creates a checkpoint, it copies the checkpoint into several copies of metadata. Running this operation to another machine can also provide fault tolerance for distributed file systems. configuration will run Secondarynamenode on a different machine

One running instance of HDFs is initiated by a $hadoop_home/bin/start-dfs.sh (or start-all.sh) script on the Namenode machine. The script starts the Namenode process on the machine running the script, and the Datanode process is started on the slaves machine, and the list of slave machines is saved in the Conf/slaves file, one machine at a time. And a SNN process is started on another machine, which is specified by the Conf/masters file. Therefore, it is important to keep in mind that the machine specified in the Conf/masters file does not say that the jobtracker or namenode process is going to run on this machine because these processes are running on launch bin/start-dfs.sh or bin/ Start-mapred.sh (start-all.sh) on the machine. So, masters This file name is very confusing, it should be called secondaries will be more appropriate. Then, use the following steps:
Write all the machines that want to run the Secondarynamenode process into the Masters file, one at a line. Modify the Conf/hadoop-site.xml file on the machine configured in the Masters file, plus the following options:
Java code <span style= "Font-size:small;" ><property> <name>dfs.http.address</name> <value>namenode.hadoop-host.com:50070</ Value> </property> </span>

Core-site.xml: Here are 2 parameters configurable, but generally we do not modify. Fs.checkpoint.period indicates how long it takes to record the image of HDFs. The default is 1 hours. Fs.checkpoint.size indicates how large the size of a record is, and the default is 64M.

Java code    <span style= "font-size: small;" ><property>     <name>fs.checkpoint.period</name>      <value>3600</value>     <description>The number of  seconds between two periodic checkpoints.     </description>    </property>      <property>     <name> fs.checkpoint.size</name>     <value>67108864</value>      <description>The size of the current edit log  (In bytes)  that triggers          a periodic checkpoint  even if the fs.checkpoint.period hasn ' t expired.     </ description>   </property>   </span>  

3, configuration check. After the configuration is complete, we need to check for success. We can determine whether the configuration was successful by looking at the file directory on the machine running Secondarynamenode. Enter JPS first to see if there is a secondarynamenode process. If present, check if there is a backup record in the corresponding directory. The following figure:

This directory usually exists under hadoop.tmp.dir/dfs/namesecondary/.

Iv. Recovery

1, the configuration is complete, how to recover. First we kill the Namenode process and then delete the data from the Hadoop.tmp.dir directory. The manufacturing master hangs out of the situation.

2. Create an empty folder at the location specified in the configuration parameter Dfs.name.dir, assign the location of the checkpoint directory to the configuration parameter Fs.checkpoint.dir, start Namenode, and add-importcheckpoint. (This is a copy of the Hadoop-0.20.2/hadoop-0.20.2/docs/cn/hdfs_user_guide.html#secondary+namenode, look at the document, there are instructions)

3. When starting Namenode, use Hadoop namenode–importcheckpoint

V. Summary

1, Secondarynamenode can be configured multiple, master files inside more than a few can be.

2, remember that if you want to restore the data is required to manually copy to the Namenode machine. is not automatic (see the recovery action written above).

3, the mirror backup cycle time can be modified, if you do not want to back up one hours, you can change the time short point. The Fs.checkpoint.period value in Core-site.xml