The road to mathematics-distributed computing-hdfs (1)

First, HDFs write data

1, the client sends the request through the Hadoop file system related API. Opens a file to be written to, and if the user has sufficient permissions, the request is sent to Namenode to establish the metadata for the file on Namenode.

2. The newly created file metadata does not associate the file with any data blocks, and the client only receives a successful response to the open file.

3. The client writes data to the stream at the API level, the data is automatically split into packets, and the packet is saved in the memory queue.

4. The client has a separate thread that reads the packet from the queue and simultaneously requests a set of Datanode lists to Namenode to write multiple copies of the next block of data.

The client connects directly to the first datanode in the list, and the Datanode is connected to the second Datanode, and eventually the client writes all the remaining packets to disk, closes the data stream, and notifies the Namenode that the file has finished writing. Throughout the process, Datanode will confirm that the received data was successfully written, the client maintains a list, records which packets have not received the message, and Datanode writes full, the client re-requests the next datanode to Namenode.

If an error occurs during the copy process, the pipeline shuts down immediately, and packets that have not received confirmation are rolled back into the queue.

This blog all content is original, if reprint please indicate source http://blog.csdn.net/myhaspl/

The road to mathematics-distributed computing-hdfs (1)