Execution Process of namenode and secondarynamenode

I. namenode details

Files include:
Fsimage: A metadata image file. Stores the namenode memory metadata information for a certain period of time.
Edits: operation log file.
Fstime: Save the last checkpoint time
These files are stored in the Linux File System.
/Data/hadoop/HDFS/Name
[[Email protected] Name] $ ls
Current image in_use.lock previous. Checkpoint
[[Email protected] Name] $ CD current/
[[Email protected] current] $ ls
Edits fsimage fstime version
If you want to use the hadoop namenode-format command to format namenode, The in_use.lock under this namenode cannot be formatted.
More in_use.lock found to be empty ----- indicates that this file is in use
Let's talk about how to enter the current folder.
There are four files with a version
Open this
Namespace ---- the command space is a relatively independent namespace
Namespaceid =-XXXX ---- indicates the ID of the file system.
The block in datanode must have a matching relationship with namenode. If the namespaceid on both sides is the same, the datanode and
An error occurs after multiple hadoop-format operations. Because the namespaceid is formatted and changed, it does not match the namespaceid in datanode. Therefore, an error occurs after the first operation.
Tip: in_use.lock: If you want to format This namenode, hadoop will find this file. If yes, it will prompt you re-format filesystem in/data/hadoop/HDFS/Name? (Y or N) to check whether the namenode has been formatted.

Ii. Execution Process of namenode and secondarynamenode

Fsimage is very important, so it was backed up.

Core-default.xml

<Property>

<Name> hadoop. tmp. dir </Name>

<Value>/tmp/hadoop-$ {user. name} </value>

<Description> a base for other temporary directories. </description>

</Property>

Hdfs-default.xml

<Property>

<Name> DFS. Name. dir </Name>

<Value >$ {hadoop. tmp. dir}/dfs/name </value>

<Description> determines where on the local filesystem the DFS Name Node

Shocould store the name table (fsimage). If this is a comma-delimited list

Of directories then the name table is replicated in all of

Directories, for redundancy. </description>

</Property>

HA solution. However, hot backup is not supported. Configuration.

(See source code)

Execution Process: Download the metadata information (fsimage, edits) from namenode, merge the two, generate a new fsimage, save it locally, and push it to namenode, reset the edits of the namenode.

It is installed on the namenode node by default, but it is not safe!

The data of datanode is not infinitely stored. It is decided to use the fsimage with namenode. Because the fsimage is stored in the memory, the memory cannot exceed TB! (The memory cannot be stored, and the fsimage is stored in the memory.) the metadata of namenode occupies the memory.

1. Add memory to namenode

2. Upload large files as much as possible.

3. sequencefile

4. Increase the block size (this depends on the environment, that is, the size of the uploaded files is designed to be large)