HDFS Concept Detailed---name node and data node

HDFS There are two types of nodes in the cluster, with managers - worker mode run, that is, a name node ( manager ) and multiple data nodes ( worker ) . The name node manages the namespace of the file system. It maintains the file system tree and all the files and index directories in the tree. This information is permanently stored on the local disk in two forms: the namespace Mirror and the edit log. The name node also records the data node where each block of each file resides, but it does not permanently save the location of the block because the information is rebuilt by the data node when the system starts.

The client accesses the entire file system on behalf of the user by interacting with the name node and data node. The client provides a file system interface similar to the POSIX ( Portable Operating System Interface ) , so users do not need to know the name node and the data node and its capabilities when programming.

The data node is a worker of the file system. They store and provide a service for locating blocks ( when called by the user or name node ), and periodically send a list of the blocks they store to the name node.

without a name node, the file system will not be available. In fact, if the machine running the name node is destroyed, all files on the filesystem will be lost because we cannot know how to reconstruct the file from the block on the data node. Therefore, it is important that the name node be able to withstand the failure, andHadoop provides two mechanisms to ensure this.

The first mechanism is to replicate files that make up the persistent state of the file system metadata. Hadoop can configure the name node to write its persisted state on multiple file systems. These write operations are synchronous and atomic in nature. The general configuration option is to write to a remote NFS Mount (Mount)while writing on the local disk.

Another possible way is to run a two-level name node, although it cannot be used as a name node. The important role of this level two name node is to periodically merge the namespace image through the edit log to prevent the editing log from being too large. This level two name node is typically run on a separate physical computer because it also requires a large amount of CPU and memory to perform the merge operation. It saves a copy of the merged namespace mirror and can be used after the name node has failed. However, the state of the two-level name node is slower than the primary node, so if the data of the master node is lost, the loss is still unavoidable. In this case, the primary name node metadata that exists on NFS is typically copied to the two-level name node and run as the new primary name node.

HDFS Concept Detailed---name node and data node