Details of how Hadoop Distributed File System HDFs works

Hadoop Distributed File System (HDFS) is a distributed file system designed to run on common hardware. HDFs is a highly fault-tolerant system that is suitable for deployment on inexpensive machines. It provides high-throughput data access and is ideal for applications on large-scale datasets. To understand the internal workings of HDFs, first understand what a distributed file system is.

1. Distributed File System
Multiple computer networks work together (sometimes called a cluster) to solve a problem like a single system, a system we call a distributed system.
Distributed File system is a subset of distributed system, which solves the problem of data storage. In other words, they are storage systems that span multiple computers. The data stored on the Distributed file system is automatically distributed across different nodes.
Distributed file systems have broad application prospects in the Big data age, providing the scalability they need to store and process ultra-large data from networks and elsewhere.

2. Separating meta data and data: Namenode and Datanode
Each file that is stored in the file system has metadata associated with it. The metadata includes the file name, the number of I nodes (Inode), the location of the data block, and so on, while the data is the actual contents of the document.
In traditional file systems, because the file system does not span multiple machines, the metadata and data are stored on the same machine.
In order to build a distributed file system so that clients are easy to use in this system and do not need to know the activity of other clients, the metadata needs to be maintained outside the client. The design concept of HDFs is to take out one or more machines to hold the metadata and let the rest of the machines save the contents of the file.
Namenode and Datanode are the two main components of HDFs. Where the metadata is stored on the Namenode, and the data is stored on the Datanode cluster. Namenode not only manages the metadata stored on HDFs, but also records things such as which nodes are part of a cluster, a file has several copies, and so on. It also decides what the system needs to do when the cluster node goes down or the data copy is lost.
Multiple copies (replica) of each piece of data stored on HDFs are saved on different servers. In essence, Namenode is the master (master server) of HDFs, and Datanode is slave (from the server).

The

3.HDFS write process
Namenode is responsible for managing the metadata of all files stored on HDFs, which confirms the client's request and records the name of the file and the Datanode collection that stores the file. It stores this information in the in-Memory file allocation table.
For example, the client sends a request to namenode that it wants to write the "Zhou.log" file to HDFs. So, its execution is shown in Process 1.
The first step: The client sends a message to Namenode, saying that the "zhou.log" file will be written. (① in 1)
Step two: Namenode sends a message to the client, asking the client to write to Datanode A, B, and D, and to contact Datanode B directly. (② in 1)
Step three: The client sends a message to Datanode B, calls it to save a "zhou.log" file, and sends a copy to Datanode A and Datanode D. (③ in 1)
Fourth: DataNode B sends a message to DataNode A, calls it to save a "zhou.log" file, and sends a copy to DataNode D. (④ in 1)
Step Fifth: DataNode a message to DataNode D, call it to save a copy of the "Zhou.log" file. (⑤ in 1)
Step sixth: DataNode D sends A confirmation message to DataNode A. (⑤ in 1)
Step seventh: DataNode a acknowledgment message to DataNode B. (④ in 1)
Step eighth: DataNode B sends a confirmation message to the client, indicating that the write is complete. (⑥ in 1)

Figure 1 HDFs write process
in the design of a distributed file system, one of the challenges is how to ensure data consistency. For HDFs, the data is considered to be complete until all datanodes to save the data confirm that they have a copy of the file. Therefore, data consistency is done at the writing stage. A client will get the same data regardless of which datanode it chooses to read from.

4.HDFS Reading Process
In order to understand the process of reading, it can be considered that a file is composed of data blocks stored on the datanode. The client views the previously written content as shown in execution Flow 2, with the following steps:
The first step: the client asks Namenode where it should read the file. (① in 2)
Step two: Namenode send the data block information to the client. (The block information contains the IP address of the Datanode that holds the copy of the file, and the block ID required by Datanode to find the data block on the local hard drive.) ) (② in 2)
The third step: the client checks the data block information, contacts the related Datanode, requests the data block. (③ in 2)
Fourth Step: Datanode Returns the contents of the file to the client, then closes the connection and completes the read operation. (④ in 2)

Figure 2 HDFs reading process
The client obtains the data block of a file from different datanode in parallel, and then joins the data blocks to complete the file.

5. Rapid recovery of hardware failures through replicas
When everything is working, Datanode sends the heartbeat message periodically to Namenode (the default is once every 3 seconds). If Namenode does not receive heartbeat information (by default, 10 minutes) during the scheduled time, it will assume that the datanode is faulty, remove it from the cluster, and start a process to recover the data. Datanode may be out of the cluster for a variety of reasons, such as hardware failure, motherboard failure, power aging, and network failure.
For HDFs, losing a datanode means losing a copy of the block of data stored on its hard disk. If there is always more than one copy at any time (default 3), the failure will not result in data loss. When a hard drive fails, HDFs detects that the number of copies stored on the hard disk is less than required, and then proactively creates the required copy to reach the full replica count status.

6. Slicing files across multiple Datanode
In HDFs, files are cut into chunks of data, usually 64MB~128MB per block of data, and then each chunk is written to the filesystem. Different blocks of data for the same file are not necessarily saved on the same datanode. The advantage of this is that when performing operations on these files, different parts of the file can be read and processed in parallel.
When the client prepares to write the file to HDFs and asks where Namenode should write the file, Namenode tells the client that the Datanode can be written to the data block. After writing a block of data, the client returns to Namenode to get a new list of Datanode and writes the next chunk of data to the Datanode on the new list.

My public number: ZHOUZXI, please scan the following two-dimensional code:

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Details of how Hadoop Distributed File System HDFs works