The role of Hadoop Secondarynamenode

1, the beginner meets the name to think that the Secondarynamenode is the Namenode backup other, or thinks they are the same. In essence, it is a snapshot of Namenode, depending on the value set in the configuration to determine how much time periodically to SPAP Namenode, record metadata and other data in Namenode.

2, if namenode damaged or lost, unable to start Hadoop at this point will be manually to intervene to restore the state of the snapshot in Secondarynamenode, which means that the cluster data will be more or less lost and some downtime, And the Secondarynamenode as an important namenode to deal with, this requires, try not to put Secondarynamede and Namenode on the same machine.

The next chapter will say how to use Secondarynamenode to restore Namenode.

English Original explanation:

Namenode appends changes to the file system to a log file (edits) on the local file system. When a namenode is started, it first reads the state of HDFs from an image file (Fsimage) and then applies the edits operation in the log file. It then writes the new HDFs State (fsimage) and begins normal operation with an empty edits file. Because Namenode merges fsimage and edits only during the start-up phase, log files can become very large after a period of time, especially for large clusters. Another side effect of a log file that is too large is that the next Namenode start will take a long time.

Secondary Namenode regularly merges fsimage and edits logs, controlling the edits log file size to a limit. Because memory requirements and NameNode are at an order of magnitude, they are usually secondary NameNode (running on separate physical machines) and NameNode running on different machines. Secondary Namenode is started by bin/start-dfs.sh on the node specified in Conf/masters.

The checkpoint process of the secondary namenode is started and is controlled by two configuration parameters:

* Fs.checkpoint.period, specifying the maximum time interval for successive checkpoints, the default value is 1 hours.
* Fs.checkpoint.size defines the maximum value of the edits log file, which, once exceeded, causes checkpoints to be enforced even if the maximum time interval to checkpoints is not reached. The default value is 64MB.

Secondary Namenode The directory structure that holds the most recent checkpoint is the same as the Namenode directory. So Namenode can read the checkpoint image on the secondary namenode when needed.

If all other historical images and edits files are lost on Namenode except for the latest checkpoints, Namenode can introduce this latest checkpoint. The following actions can be implemented for this function:

* Create an empty folder at the location specified in configuration parameter dfs.name.dir;
* Assign the location of the checkpoint directory to the configuration parameter fs.checkpoint.dir;
* Start Namenode, and add-importcheckpoint.

Namenode reads the checkpoint from the Fs.checkpoint.dir directory and saves it in the Dfs.name.dir directory. If there is a valid image file under the Dfs.name.dir directory, Namenode will fail to start. Namenode will check the consistency of the image file in the Fs.checkpoint.dir directory, but will not change it.