Introduction to collaboration and communication between Namenode, Datanode, and client in Hadoop

<ignore_js_op>

1) NameNode, Datanode and client
Namenode can be seen as a manager in a distributed file system, primarily responsible for managing the file system's namespace, cluster configuration information, andStorage block replication, and so on. Namenode will store the file system's Meta-data in memory, which mainly includes the file information, the corresponding file block of each file, and the information of each file block in Datanode.
Datanode is the basic unit of file storage, which stores blocks in the local file system, preserves block meta-data, and periodically sends all existing block information to Namenode.
The client is the application that needs to obtain the Distributed File System files.



2) File Write
The client initiates a request to the Namenode to write the file.
Namenode is returned to the client based on the file size and file block configurationManage part of the Datanode information.
The client divides the file into blocks, which are written sequentially to each Datanode block according to the Datanode address information.

3) file read
The client initiates a request to the Namenode to read the file.
Namenode returns the datanode information for the file store.
The client reads the file information.

--------------------------------------------------------------------------------------------------------------- -------------------------------------------------

Introduction of Communication methods:

In the Hadoop system, the correspondence between master/slaves/client is:
Master---namenode;
Slaves---datanode;
Client---dfsclient;
What is the way to communicate, here from a general introduction:
In brief:
Between the client and the Namenode is through RPC communication;
Between Datanode and Namenode is through RPC communication;
Between the client and Datanode is through a simple socket communication.
Just unplug the Dfsclient code, you can see it has a member variable public final clientprotocolnamenode;
And then pull the Datanode code, you can see it also has a member variable public datanodeprotocolnamenode

Article turned from: http://www.aboutyun.com/thread-6794-1-1.html

Introduction to collaboration and communication between Namenode, Datanode, and client in Hadoop