Flow interpretation of Read and write data on HDFs

Read the file

The file read process is as follows:

1) Explain a

The client opens the file with the FileSystem open () function.

Distributedfilesystem uses RPC to call the metadata node to get the data block information of the file.

For each chunk, the metadata node returns the address of the data node that holds the data block.

Distributedfilesystem returns Fsdatainputstream to the client, which is used to read the data.

The client calls the stream's read () function to begin reading the data.

The Dfsinputstream connects the closest data node that holds the first chunk of this file.

Data is read from the database node to the client.

When this block of data is read, Dfsinputstream closes the connection to this data node and then connects to the nearest data node of the next block of data for this file.

When the client has finished reading the data, call Fsdatainputstream's close function.

In the process of reading data, if the client has an error communicating with the data node, it attempts to connect to the next data node that contains the data block.

The failed data node is logged and is no longer connected.

2) explain two

Initiate RPC requests to remote Namenode using the client development library provided by HDFS;

Namenode returns a partial or all block list of files as appropriate, and returns the Datanode address of the block copy for each block,namenode;

The Client development library selects the Datanode closest to the client to read the block;

After reading the current block data, close the current Datanode connection and find the best datanode for reading the next block;

When the block of the list is read and the file read is not finished, the client development library continues to fetch the next block list to Namenode.

A block is checksum verified, and if an error occurs while reading the Datanode, the client notifies Namenode and continues reading from the next Datanode that owns the block copy.

Write to File

The process of writing to a file is more complex than reading:

1) Explain a

The client calls create () to make the file

Distributedfilesystem uses RPC to call the metadata node to create a new file in the file system's namespace.

The metadata node first determines that the file does not exist, and the client has permission to create the file, and then creates a new file.

Distributedfilesystem returns Dfsoutputstream, which the client uses to write data.

The client begins to write the data, dfsoutputstream the data into chunks, and writes it to the data queue.

The data queue is read by data streamer and notifies the metadata node to allocate data nodes for storing
Hebei University of Technology--software engineering and Theoretical Laboratory Editor: Shrimp 25
Creation time: 2012/3/6 modified: 2012/3/8 modified: 0
Hebei University of Technology--software engineering and Theoretical Laboratory Editor: Shrimp 26
Block (the default copy of 3 blocks per block). The assigned data node is placed in a pipeline.

Data Streamer writes a block to the first data node in the pipeline. The first Data node sends a block of data to the second data node. The second data node sends the data to a third data node.

Dfsoutputstream holds an ACK queue for the emitted data block, waiting for the data node in the pipeline to tell the data has been successfully written.

If the data node fails during the write process:

Close pipeline to the beginning of the data queue for the block of ACK in the queue.

The current data block is given a new flag by the metadata node in the data node that is already written, and the error node restarts to detect that its data block is obsolete and is deleted.

The failed data node is removed from the pipeline, and the other data block is written to the other two data nodes in the pipeline.

The metadata node is notified that the block is insufficient in number of copies and will create a third backup in the future.

The close function of the stream is called when the client ends writing the data. This operation writes all data blocks to the data node in pipeline and waits for the ACK queue to return successfully. Finally notifies the metadata node that the write is complete.
2) explain two

Initiate RPC requests to remote Namenode using the client development library provided by HDFS;

Namenode will check if the file to be created already exists, if the creator has permission to do so, the success will create a record for the file, otherwise it will throw an exception to the client;

When the client begins writing to the file, the development library splits the file into multiple packets, manages the packets internally as a "data queue", and applies a new blocks to Namenode, Gets the appropriate list of datanodes to store replicas, depending on the settings for replication in Namenode.

begins to write packet to all replicas in the form of pipeline (pipe). The development library writes the packet to the first datanode in a stream, and the Datanode stores the packet and passes it to the next pipeline in this Datanode, until the last Datanode, This method of writing data is in the form of pipelining.

After the last Datanode is successfully stored, it returns an ACK packet, which is passed to the client in pipeline, maintains an "ACK queue" inside the client's development library, and receives the Ackpacket returned by Datanode from the "ACK queue "Remove the corresponding packet.

If a datanode fails during transmission, the current pipeline will be closed, the failed datanode will be removed from the current pipeline, and the remaining blocks will continue to be the remaining DAtanode continues to be transmitted as pipeline, while Namenode assigns a new datanode, maintaining the number of replicas settings.

Reference documents:

Http://blog.sina.com.cn/s/blog_9402246001013ktm.html

Http://www.cnblogs.com/xia520pi/archive/2012/05/28/2520813.html

Flow interpretation of Read and write data on HDFs