Detailed working principle of Hadoop Distributed File System HDFS,

Source: Internet
Author: User

Detailed working principle of Hadoop Distributed File System HDFS,

Hadoop Distributed File System (HDFS) is a distributed file system designed to run on common hardware. HDFS is a highly fault tolerant system and is suitable for deployment on cheap machines. It provides high-throughput data access and is very suitable for applications on large-scale datasets. To understand the internal working principle of HDFS, we must first understand what a distributed file system is.

1. Distributed File System
Multiple computers work collaboratively on the Internet (sometimes called a cluster) to solve a problem like a single system. Such a system is called a distributed system.
Distributed File systems are a subset of distributed systems. They solve the problem of data storage. In other words, they are stored across multiple computers. Data stored in the Distributed File System is automatically distributed across different nodes.
Distributed File systems have broad application prospects in the big data era. They provide the necessary scalability to store and process ultra-large-scale data from networks and other places.

2. Separated metadata and data: NameNode and DataNode
Each file stored in the file system has associated metadata. Metadata includes the file name, inode number, and data block location, while data is the actual content of the file.
In traditional file systems, because file systems do not span multiple machines, metadata and data are stored on the same machine.
In order to build a distributed file system, make the client easy to use in this system, and do not need to know the activities of other clients, metadata needs to be maintained outside the client. HDFS is designed to use one or more machines to store metadata and allow the remaining machines to store the file content.
NameNode and DataNode are two main components of HDFS. Metadata is stored on NameNode, and data is stored on the DataNode cluster. NameNode not only manages metadata stored on HDFS content, but also records some things, such as which nodes are part of the cluster and several copies of a file. It also determines what the system needs to do when the cluster node goes down or the data copy is lost.
Each data segment stored on HDFS has multiple copies (replica) stored on different servers. In essence, NameNode is the Master (Master server) of HDFS and DataNode is Slave (Slave server ).

3. HDFS write process
NameNode is responsible for managing the metadata of all files stored on HDFS. It confirms client requests and records the file name and the DataNode set for storing the file. It stores the information in the file allocation table in the memory.
For example, a client sends a request to NameNode, saying that it wants to write the "zhou. log" file to HDFS. Then, the execution process 1 is shown. Specifically:
Step 1: the client sends a message to NameNode, saying that the "zhou. log" file should be written. (① In 1)
Step 2: NameNode sends A message to the client, asking the client to write data to DataNode A, B, and D, and directly contact DataNode B. (② In 1)
Step 3: the client sends A message to DataNode B and asks it to save A file named "zhou. log" and send A copy to DataNode A and DataNode D. (③ In 1)
Step 4: DataNode B sends A message to DataNode A, asking it to save A file named "zhou. log" and send A copy to DataNode D. (4 in 1)
Step 5: Send A message to DataNode D and ask it to save A file named "zhou. log. (⑤ In 1)
Step 6: Send A confirmation message to DataNode. (⑤ In 1)
Step 7: Send A confirmation message to DataNode B. (4 in 1)
Step 8: Send a confirmation message to DataNode B to the client, indicating that the write is complete. (6 in 1)

Figure 1 HDFS write process
One of the challenges in the Design of Distributed File Systems is how to ensure data consistency. For HDFS, data is considered to have been written until all the DataNodes that want to save the data confirm that they all have copies of the file. Therefore, data consistency is completed at the write stage. A client obtains the same data no matter which DataNode it chooses to read.

4. HDFS read Process
To understand the read process, a file is considered to be composed of data blocks stored on DataNode. The client can view the execution process of the previously written content. The specific steps are as follows:
Step 1: The client asks where NameNode should read the file. (① In 2)
Step 2: NameNode sends the data block information to the client. (The data block information includes the IP address of DataNode that saves the copy of the file, and the ID of the data block required for DataNode to search for the data block on the local hard disk .) (② In 2)
Step 3: the client checks the data block information, contacts the related DataNode, and requests the data block. (③ In 2)
Step 4: DataNode returns the file content to the client, closes the connection, and completes the read operation. (4 in 2)

Figure 2 HDFS read Process
The client obtains data blocks of a file from different DataNode in parallel, and then connects these data blocks to form a complete file.

5. Fast Recovery of hardware faults through copies
When everything works normally, DataNode periodically sends heartbeat information to NameNode (default: every 3 seconds ). If NameNode does not receive heartbeat information within the specified period (10 minutes by default), it will consider that DataNode has a problem, remove it from the cluster, and start a process to restore data. DataNode may be removed from the cluster for multiple reasons, such as hardware faults, motherboard faults, power aging, and network faults.
For HDFS, the loss of a DataNode means that a copy of the data block stored on its hard disk is lost. If more than one replica exists at any time (three by default), failure will not cause data loss. When a hard disk fails, HDFS detects that the number of copies of data blocks stored on the hard disk is lower than the requirement, and then creates the required copies to reach the full number of copies.

6. Split files across multiple DataNode
In HDFS, files are split into data blocks. Generally, each data block is 64 MB ~ 128 MB, and each data block is written to the file system. Different data blocks of the same file may not be stored on the same DataNode. The advantage of this is that when performing operations on these files, different parts of the files can be read and processed in parallel.
When the client is preparing to write files to HDFS and asks where NameNode should write the files, NameNode will tell the client which DataNode can write data blocks. After writing a batch of data blocks, the client returns to NameNode to obtain the new DataNode list and writes the next batch of data blocks to the DataNode in the new list.

My public account: zhouzxi. Please scan the following QR code:

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.