Hadoop learning note _ 6_distributed File System HDFS -- namenode Architecture

Distributed File System HDFS-namenode architecture namenode

Is the management node of the entire file system.

It maintains the file directory tree of the entire file system [to make retrieval faster, this directory tree is stored in memory],

The metadata of the file/directory and the data block list corresponding to each file.

Receives user operation requests.

Hadoop ensures the robustness of namenode and is not prone to death. the file directory tree and the metadata of the file/directory are stored in the hard disk in the final form. However, when hadoop is running, it needs to be loaded into the memory.

 

Files include:

Fsimage: A metadata image file. Stores the namenode memory metadata information for a certain period of time.

Edits: operation log file.

Fstime: Save the last checkpoint time

These files are stored in the Linux File System.

 

1. View hdfs-default.xml File Content

<Property> <Name> DFS. name. dir </Name> <value >$ {hadoop. TMP. dir}/dfs/name </value> <description> determines where on the local filesystem the DFS Name node shocould store the name table (fsimage: File System image) (specify the location where namenode stores the name table on the local machine ). $ {hadoop. TMP. dir} will be replaced with a value in the core-site.xml file, see, if this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. for data security, you can write this value into a list of directories separated by commas (,) and specify multiple storage locations of fsimage to ensure security, such as $ {hadoop. TMP. dir}/dfs/Name, newdir0, newdir1 but it is not modified here, it must be copied to the hdfs-site.xml file, and do not add spaces before and after the comma, the more dispersed these folders, the better </description> </property>

2. Enter core-site.xml

<property>  <name>hadoop.tmp.dir</name>  <value>/tmp/hadoop-${user.name}</value>  <description>A base for other temporary directories.</description></property>

3. Return the hdfs-default.xml File

<Property> <Name> DFS. name. edits. dir </Name> <value >$ {DFS. name. dir} </value> <description> determines where on the local filesystem the DFS Name node shocould store the transaction (edits) file (transaction file, used to save the Upload File transaction process (Transaction) file). For the specific execution process, see secondary namenode. if this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. default value is same as DFS. name. dir </description> </property>

 

Secondarynamenode

Execution Process: Download the metadata information (fsimage, edits) from namenode, merge the two, generate a new fsimage, save it locally, and push it to namenode, reset the edits of the namenode.

Once the fsimage data in namenode is lost or damaged, you can call the fsimage backup in secondaryname to restore the data. However, these data files do not contain unmerged data files. The function is similar to restore origin in windows.

Secondarynamenode is installed on the namenode node by default, but this is not safe!

 

Merging principle: