Hadoop learning-secondarynamenode

When hadoop is started, a secondarynamenode process is started on the namenode node by default, which can be viewed using JSP commands. Secondarynamenode is literally a hot standby process of namenode. Actually not. secondarynamenode is an integral part of the HDFS architecture. It is not a slave node when a metadata node is faulty. It is responsible for different tasks.

 

1. Purpose of the secondarynamenode node:

To put it simply, the main function of the secondarynamenode node is to periodically merge the namespace image file and the modification log of the metadata node to prevent the log file from being too large.

 

 

To understand the functions of secondarynamenode, first, let's take a look at the main work of namenode:

2. Main Work of the namenode node:

One of the main functions of namenode is to manage the namespace of the file system, which stores the metadata of all files and file directories in a file system tree. To ensure the interaction speed, namenodeIn MemorySave the metadata information, but also save the information to the hard disk.Persistent StorageIs saved as the namespace image file (fsimage) and the log file (edits ). Is the file directory structure on the namenode node:

The fsimage file, that is, the namespace image file, is the checkpoint of the metadata in the memory on the hard disk. It is a serialized format and cannot be directly modified on the hard disk.

With these two files, hadoop can restore the status of these two files upon restart. fsimage is equivalent to a checkpoint. Therefore, when hadoop is restarted, two files are required: fsimage + edits, first, load the latest checkpoint metadata information from fsimage to the memory, and then execute edits to modify the operations in the log file one by one to restore the final state before the restart.

The persistence process of hadoop is to save the last operation after checkpoint to modify the log file edits.

 

One problem occurs here is that edits will become larger and larger with the increase of time, which will take a long time to recover from the restart according to the operations recorded in edits.Therefore, hadoop uses secondarynamenode, which is used to help metadata nodes to checkpoint the metadata information in the memory to the hard disk.

 

3. secondarynamenode workflow:

• The secondarynamenode node notifies the namenode node to generate a new log file. Later logs are written to the new log file.

• The secondarynamenode node uses http get to obtain the fsimage file and old log file from the namenode node.

• The secondarynamenode node loads the fsimage file into the memory, performs operations in the log file, and generates a new fsimage file.

• The secondarynamenode node sends the new fsimage file back to the namenode node using http post.

• The namenode node can replace the old fsimage file and the old log file with the new fsimage file and the new log file (generated in the first step), and then update the fstime file, the time when the checkpoint is written.

• In this way, the fsimage file in the namenode node stores the latest checkpoint metadata information, and the log file starts again, so it will not become very large.

• The flowchart is as follows:

 

4. secondarynamenode runs on anotherOn a non-namenode Machine

By default, the secondarynamenode process runs on the machine on the namenode node. If this machine fails or goes down, it is a great disaster to restore the HDFS file system. The better way is: configure the secondarynamenode process to run on another machine.The reason for running the SNN process on a non-namenode machine is as follows:

Scalability: to create a new HDFS snapshot, You need to copy all the metadata information loaded to the memory in namenode. This operation requires the same memory as the memory occupied by namenode, the memory allocated to the namenode process is actually a limitation on the HDFS file system. If the distributed file system is very large, the memory of the namenode machine may be occupied by all namenode processes.

Fault Tolerance: When SNN creates a checkpoint, it copies the checkpoint to several copies of metadata. Running this operation on another machine also provides fault tolerance for distributed file systems.

 

 

5. Configure to run secondarynamenode on another machine:

A running instance of HDFS is started through the $ hadoop_home/bin/start-dfs.sh (or start-all.sh) script on the namenode machine. This script starts the namenode process on the machine that runs the script, and the datanode process is started on the server Load balancer instance. The list of Server Load balancer instances is saved in the conf/Server Load balancer file, with one machine in one row. A secondarynamenode process will be started on another machine, which is specified by the conf/masters file. Therefore, note that the machine specified in the conf/masters file does not mean that the jobtracker or namenode process must run on this machine, because these processes are running on launch bin/start-dfs.sh or bin/start-mapred.sh (start-all.sh) machines. Therefore, the name of the masters file is very confusing and should be called secondaries. Follow these steps:

 

1. Write all the machines that want to run the secondarynamenode process to the masters file, one row at a time.

2. Modify the conf/hadoop-site.xml file on the machine configured in the Masters file with the following options:

   <property>    <name>dfs.http.address</name>    <value> :50070</value> </property>

3.  

Hadoop learning-secondarynamenode