Mapreduce (15th): source code analysis for reading files from HDFS

Take the map task as an example to read text data:

1) linerecordreader is responsible for locating file segmentation and encapsulating each row of content for user map tasks. Each time the row is located at a position not 0 in the file, it reads one more row, because when the previous row processes the data before the position, it reads and processes the entire row.

 

2) linereader is responsible for reading each row of data in the input stream of the file to be accessed, and only implements the read logic for each row.

3) dfsdatainputstream encapsulates the implementation of dfsinputstream and directly calls the dfsinputstream interface.

4) dfsinputstream is responsible for managing the block of the accessed file, providing the location of the block and the interface for reading the block data stream.

Locate the block in which the data is read, open the socket connection to the block, and then read data from the remoteblockreader that encapsulates the socket:

Open the socket connection to the host based on the host where the block is located:

After obtaining the block information, start to get the data stream:

5) remoteblockreader encapsulates the datainputstream input stream for obtaining datanode from the open socket to start reading file data. The file data is encapsulated by datanode and transmitted to the client once per chunk. The client checks data integrity for each chunk.

Provides an interface for reading data streams from the socket to the upper layer:

Start reading data from chunks:

 

Read per Chunk:

6) block transmission read Protocol

First, after the client opens the connection, enter the block information to datanode:

Then, datanode sends data streams to dfsclient based on each chunk. The chunk data format is listed in the order of Data Reading:

Mapreduce (15th): source code analysis for reading files from HDFS