Hadoop Source Code Analysis (17) DataNode

after clearing the surrounding obstacles, we can begin to analyze class Datanode. The class diagram is as follows:
 

Publipublic class DataNode extends configured
implements Interdatanodeprotocol, Clientdatanodeprotocol, Fsconsta nts, Runnable
given the inheritance of DataNode, we find that the DataNode implements two communication interfaces, where Clientdatanodeprotocol is used for and client
The Interactive, Interdatanodeprotocol, is the communication interface between the Datanode we mentioned earlier. Ipcserver (lower left of the class diagram) is Datanode
A member variable that initiates an IPC service so that DataNode can provide clientdatanodeprotocol and Interdatanodeprotocol
the Force.
let's start with the main function. This function is simple, calls the Createdatanode method, and waits for the datanode thread to end. Createdatanode
First Call Instantiatedatanode to initialize Datanode, and then execute Rundatanodedaemon. Rundatanodedaemon will register with Namenode, as
successful, the DataNode thread was started, and DataNode began to work.
the method of initializing the Datanode Instantiatedatanode reads the configuration file Datanode requires, while reading the configured storage directory (there may be multiple,
See Storage's Discussion section), and then send these two parameters to Makeinstance, Makeinstance will first check the directory (exist, is the directory, readable, writable),
then call:
New DataNode (conf, dirs);
The control flow is then on the constructor. The constructor calls Startdatanode, completing and DataNode related initialization work (note that the DataNode work line
The process does not start in this function). The first is to initialize a bunch of configuration parameters, what Namenode address, socket parameters, and so on. Then, request a namenode to the
(datanodeprotocol.versionrequest), and check that the Namespaceinfo and local versions returned are consistent.
the next step in the normal situation is to check the state of the file system and do the necessary recovery, initializing the Fsdataset (by this time, the storage and data members in the above image are changed
(The volume has been initialized).
then, find a port and create a dataxceiverserver (start in the Run method), create the Datablockscanner (start in offerservice as needed,
start only once), create the Httpserver on the Datanode, and start the ipcserver. This ends the initialization of the Datanode related work.
before starting the DataNode worker thread, DataNode needs to register with Namenode. The registration information has been constructed at the time of initialization, including Dataxceiverserver
Port, ipcserver port, file layout version number and other important information. After successful registration, you can start the Datanode thread.
DataNode Run method, there are two options in the loop, upgrade (temporarily not discussed)/normal operation. Let's look at the Offerservice method of normal work. Offerservice
is also a cycle, in the loop, Offerservice will periodically send heartbeat to Namenode, report the change of block status in the system, report Datanode now manage
block state. When sending heartbeat and block status reports, NameNode will return some commands, DataNode will execute them.
the heartbeat processing is relatively simple and is sent at heartbeatinterval intervals.
The Block status change report takes advantage of the information saved in Receivedblocklist and delhints two lists. Receivedblocklist shows that in this
DataNode successfully creates a new block of data, while delhints is the node that can delete the data block. As in Dataxceiver's replaceblock, there are calls:
Datanode.notifynamenodereceivedblock (block, SourceID)
This indicates that DataNode has received a block,sourceid on the corresponding block from the SourceID can be deleted (this scenario is present when the system needs
when doing load balancing, the Block is copied between Datanode).
The Block status change report is reported through Namenode.blockreceived.
The Block status report is also relatively simple and is sent at blockreportinterval intervals.
Heartbeat and block status reports can return commands, which is the only way Namenode Datanode initiate a request first. Let's take a look at all the commands:
Dna_transfer: Copying data blocks to other Datanode
dna_invalidate: Deleting data blocks (simple method)
Dna_shutdown: Close Datanode (simple method)
Dna_register:datanode Re-registration (simple method)
dna_finalize: Submit Upgrade (Simple method)
dna_recoverblock: Recovering data blocks
copying data blocks to other Datanode is performed by the Transferblocks method. Note that the returned command can contain more than one block of data, each of which can contain multiple
Destination address. The Transferblocks method initiates a DataTransfer thread for each block that transmits the data.
DataTransfer is an internal class of Datanode that uses the Op_write_block write block operations we described earlier to send data to multiple targets.
recovery data block and Namenode lease (lease) recovery is concerned, we will discuss later.

Hadoop Source Code Analysis (17) DataNode