Namenode and secondary Namenode communication model

Namenode appends changes to the file system to a log file edits on the local file system. when a namenode is started, it first reads the state of HDFs from an image file (Fsimage) and then performs the edit operation in the log file. The new HDFs state is then written to Fsimage, and the well uses an empty edits file to begin normal operation . Because Namenode merges fsimage and edits only during the startup phase, the log files can become very large over time, especially for large clusters. Another side effect of a log file that is too large is that the next Namenode start will be very long communication between the world, Namenode and secondary namenode.

As shown in. The communication between Namenode and secondary namenode data is using the HTTP protocol , secondary Namenode periodically merges fsimage and edits logs , Control the size of the edits log file to a limit. Because memory requirements and Namenode are at an order of magnitude, typically secondary namenode and Namenode run on different machines. secondary Namenode is started by bin/start-dfs.sh on the node specified in Conf/masters.
The checkpoint process for secondary namenode is initiated by the following two configuration parameters:
FS.CHECKPOINT.PERIOD Specifies the maximum time interval for successive checkpoints, and the default value is 1 hours.

Fs.checkpoint.size defines the maximum value of the log file, which, once exceeded, causes the checkpoint to be enforced (even if it does not reach the checkpoint's maximum interval), the default value is 64MB.

Secondary Namenode The directory structure that holds the most recent checkpoint is the same as the Namenode directory. So Namenode can read the checkpoint image on the secondary namenode when needed.
If all other historical images and edits files are lost on Namenode except for the latest checkpoints, Namenode can introduce this latest checkpoint. The following actions can be implemented for this function:
L) Create an empty folder at the location specified in configuration parameter dfs.name.dir.
2) Assign the location of the checkpoint daily to the configuration parameter fs.checkpoint.dir.
3) Start Namenode, plus-importcheckpoint.

Namenode will read the checkpoint from the Fs.checkpoint.dir directory and save it on the Dfs.name.dir day record. If there is a valid image file under the Dfs.name.dir directory, Namenode will fail to start. Namenode will check the consistency of the image file in the Fs.checkpoint.dir directory, but will not change it.

Namenode and secondary Namenode communication model