Principle Analysis of HDFS File Reading in Hadoop

Principle Analysis of HDFS File Reading in Hadoop

The previous article briefly introduced some logic and simple principles of Hadoop file storage (see). When writing data, you must read the analysis data, the following describes the logic and simple principle of File Reading in hdfs.

Step 1: like writing a file, the client will first call the open () method of the DistributedFilesyStem object to open the file. What this method does is: distributed FilesyStem calls namenode through RPC protocol connection. namenode stores all file namespaces, that is, the address where files are stored in datanode, first, we get the location of the file header to be read. There are many data node copies in the block. hadoop will find a node closest to the client according to certain standards, in this case, a FSData InputStream is returned; otherwise, ioexception is returned.

Step 2: the client will read the returned file input stream. In this case, the datanode stored in the file header will find the other datanode closest to itself in these blocks and establish a link. The client continues to read, until it is read to the end of the block. Find the address of the next block from namenode and find the best file node location. Repeat the above actions. After reading is complete, the file input stream will call the close method to close the stream,

Next we will discuss the Exception Handling Mechanism:

What should I do if the client encounters an error block while reading the data stream?

If the client encounters an exception block, the client will record the block and try to read the block closest to the block, and will not read the damaged block again.

At the same time, the client will verify the received data checksum. If a damaged block is found, it will report to the Name node before the client tries to read a copy of a block from another data node.

As we have mentioned before, how does one implement hadoop's mechanism of finding the nearest block or node?

We all know. In big data storage, the main factor limiting efficiency is bandwidth.
Hadoop regards the network as a tree, and the distance between the two nodes is the sum of the common ancestor closest to them.
For each of the following scenarios, the available bandwidth is reduced in sequence:
Processes on the same node
Different nodes on the same rack
Nodes on different racks of the same data center
Nodes in different data centers

How HDFS reads and writes in Hadoop work

Copy local files to HDFS

Download files from HDFS to local

Upload local files to HDFS

Common commands for HDFS basic files

Introduction to HDFS and MapReduce nodes in Hadoop