"Go" fsimage and edits merge implementations in Hadoop 1.x

During the Namenode run, all of the update operations for HDFS are written directly to edits, and the edits file will become large over time, although this has no effect on the Namenode runtime, but we know that when the Namenode restarts, Namenode all the content inside the fsimage into memory first, and then one by one to execute the edits in the record, when the edits file is very large, it will cause the Namenode start operation is very slow, and during this time the HDFS system in Safe mode, This is clearly not required by the user. Can you make the edits file smaller when the Namenode is running? In fact, it is possible, this article is mainly forHadoop 1.x version that explains how it merges edits and fsimage files,hadoop 2.x version edits and fsimage file merging are different.
users who have used Hadoop should know that there is a secondarynamenode process in Hadoop, and it is easy to think of it as a namenode hot-standby process in terms of name. In fact, the real situation is not so. Secondarynamenode is an integral part of the HDFS architecture, it is used to save namenode in the metadata of HDFs information backup, and reduce the time of Namenode restart set! It is common to run Secondarynamenode alone on a single machine, so how does Secondarynamenode namenode restart time? Take a look at the work of Secondarynamenode:
(1) Secondarynamenode periodically and namenode communication, requesting it to stop using edits files, Temporarily writes a new write operation to a new file edit.new up, this operation is instantaneous completes, the upper layer writes the log function completely to feel the difference;
(2) Secondarynamenode via HTTP The Get method obtains the Fsimage and edits files from the Namenode and downloads them to the local directory;
(3) Secondarynamenode loads the downloaded fsimage into memory, And then a one-by-one implementation of the edits file in the update operation, so that in-memory fsimage is kept up-to-date, this process is edits and fsimage file merging;
(4) Secondarynamenode after performing (3) operation, The new Fsimage file is sent to the Namenode node by post, and
(5) Namenode will replace the old Fsimage file with the new fsimage received from Secondarynamenode. At the same time replace the edit.new edits file, through this process edits become smaller! The execution of the entire process can be illustrated by the following diagram:
in the (1) step, we talked about Secondarynamenode regular and namenode communication, which needs to be configured, can be configured through Core-site.xml, the following is the default configuration:

1 <property>2   <name>fs.checkpoint.period</name>3   <value >3600</value>4<description> The number of   seconds between, periodic checkpoints. 5   </description>6 </property>

In fact, if the Fs.checkpoint.period configuration time has not expired, we can also determine the current edits size to trigger a merge operation, can be configured by the following

1 <property>2   <name>fs.checkpoint.size</name>3   <value >67108864</value>4<description> the size of the current   edit log (in bytes) that Triggers5        if the fs.checkpoint.period hasn ' t expired.  6   </description>7 </property>

When the edits file size exceeds the above configuration, a merge is made even if the fs.checkpoint.period is not yet there. By the way, the Secondarynamenode downloaded fsimage and edits temporarily stored paths can be configured with the following properties:

1<property>2<name>fs.checkpoint.dir</name>3<value>${hadoop.tmp.dir}/dfs/namesecondary</value>4<description>determines where on the local filesystem the DFS secondary5 name node should store the temporary images to merge.6If Thisis a comma-delimited list of directories then the image is7Replicated in all of the directories forredundancy.8</description>9</property>Ten  One<property> A<name>fs.checkpoint.edits.dir</name> -<value>${fs.checkpoint.dir}</value> -<description>determines where on the local filesystem the DFS secondary the name node should store the temporary edits to merge. -If Thisis a comma-delimited list of directoires then teh edits are -Replicated in all of the Directoires forredundancy. - Default value is same as Fs.checkpoint.dir +</description> -</property>

As we can see from the above description, Secondarynamenode is not a hot preparation for namenode, it simply merges fsimage and edits. Its fsimage is not up-to-date, because when he downloads fsimage and edits files from Namenode, the new update operation has been written to the Edit.new file. And these updates are not synced to the Secondarynamenode! Of course, if the namenode in the fsimage really problem, still can use the fsimage in Secondarynamenode to replace Namenode, although it is not the latest fsimage, But we can reduce the loss to a minimum!
In Hadoop 2.x by configuring Journalnode to achieve high availability of Hadoop, so that the main is namenode on the fsimage and edits are up to date, any time as long as a namenode hang, It can also make the fsimage in the cluster up to date! Reprinted from: http://www.iteblog.com/archives/969

"Go" fsimage and edits merge implementations in Hadoop 1.x