The role of Hadoop Secondarynamenode __mapreduce

1, beginners meet the name of Secondarynamenode think that the Namenode is a backup of the other, or that they are the same. Essentially, it is a snapshot of Namenode, depending on the value set in configuration to determine how much time periodically to SPAP Namenode, and to record namenode and other data in metadata.

2, if namenode damaged or lost, unable to start Hadoop at this time will be manually to intervene to restore the snapshot in the Secondarynamenode state, which means that the cluster data will be more or less lost and some downtime, And the Secondarynamenode as an important namenode to deal with, which requires, try not to put Secondarynamede and Namenode on the same machine.

The next article will say how to use Secondarynamenode to restore Namenode.

English Original explanation:

Namenode will append changes to the file system to a log file (edits) on the local file system. When a namenode is started, it first reads the state of the HDFs from an image file (Fsimage) and then applies the edits operation in the log file. It then writes the new HDFs state to (fsimage) and begins normal operation with an empty edits file. Because Namenode only merges fsimage and edits during the startup phase, log files can become very large over time, especially for large clusters. Another side effect of the log file that is too large is that the next Namenode startup can take a long time.

Secondary Namenode periodically merges fsimage and edits logs, controlling the edits log file size to a limit. Because the memory requirements and Namenode are on an order of magnitude, usually secondary namenode (running on a separate physical machine) and Namenode run on different machines. Secondary Namenode is started by bin/start-dfs.sh on the node specified in Conf/masters.

The checkpoint process for secondary namenode is started and is controlled by two configuration parameters:

* Fs.checkpoint.period, specifies the maximum interval of two consecutive checkpoints, with a default value of 1 hours.
* Fs.checkpoint.size defines the maximum value of the edits log file, which causes the checkpoint to be enforced even if the maximum time interval for the checkpoint is not exceeded. The default value is 64MB.

Secondary Namenode The directory that holds the most recent checkpoint is the same as the Namenode directory structure. So Namenode can read the checkpoint mirror on the secondary namenode when needed.

If all other historical mirrors and edits files are lost on Namenode except for the most recent checkpoints, Namenode can introduce this latest checkpoint. This functionality can be achieved by doing the following:

* Create an empty folder at the specified location in the configuration parameter dfs.name.dir;
* Assigning the location of the checkpoint directory to the configuration parameter fs.checkpoint.dir;
* Start Namenode and add-importcheckpoint.

The

Namenode reads the checkpoint from the Fs.checkpoint.dir directory and saves it in the Dfs.name.dir directory. If a legitimate mirrored file is available under the Dfs.name.dir directory, Namenode will fail to start. Namenode checks the consistency of the mirrored file in the Fs.checkpoint.dir directory, but does not change it.