Analysis of hadoop heartbeat Mechanism

The heartbeat mechanism is probably like this:
1) when the master node is started, an IPC server will be opened there.
2) When slave is started, it will connect to the master and send a "Heartbeat" to the master every three seconds to tell its status information. Then, the master also returns a heartbeat value, send commands to the slave node.

2. Find the heartbeat code

For namenode and datanode, In the offerservice method of datanode, the heartbeat code is sent to namenode every three seconds:

 

/*** Main loop for the datanode. runs until shutdown, * Forever calling remote namenode functions. */Public void offerservice () throws exception {... /// now loop for a long time .... // while (shouldrun) {try {long starttime = now (); // every so often, send heartbeat or block-Report /// if it takes 3 seconds, send heartbeat to namenode if (starttime-lastheartbeat> heartbeatinterval) {// All heartbeat mesval Es include following info: // -- datanode name // -- data transfer port // -- total capacity // -- bytes remaining // lastheartbeat = starttime; datanodecommand [] cmds = namenode. sendheartbeat (dnregistration, Data. getcapacity (), Data. getdfsused (), Data. getremaining (), xmitsinprogress. get (), getxceivercount (); // note that in the above line of code, "Send heartbeat" is actually a method to call namenode ?? Mymetrics. heartbeats. INC (now ()-starttime); // log.info ("just sent heartbeat, with name" + localname); // process the return value of heartbeat (the instruction that namenode sends to datanode) if (! Processcommand (cmds) continue;} // many codes are omitted here...} // while (shouldrun)} // offerservice

 

The code above is nothing strange if it is a single-Host Program. However, this is a hadoop cluster! Datanode and namenode run on two different machines (or two JVMs! The datanode machine calls the namenode method directly! How is this implemented? Is it the legendary RMI ??

Next we will mainly analyze the details of this method call.

 

3. Details of Heartbeat Layer 1: How does datanode obtain the namenode object?

First, there is a namenode member variable in the datanode class:

public class DataNode extends Configured     implements InterDatanodeProtocol, ClientDatanodeProtocol, FSConstants, Runnable {  ...  public DatanodeProtocol namenode = null;  ... }

 

The following is the definition of the namenode class:

public class NameNode implements ClientProtocol, DatanodeProtocol,                                 NamenodeProtocol, FSConstants,                                 RefreshAuthorizationPolicyProtocol {  ... }

 

Note: namenode implements the datanodeprotocol interface. The datanodeprotocol interface defines the communication method between namenode and datanode.

So how does the datanode class obtain the reference of the namenode class?

Code that assigns a value to the namenode variable on the datanode end:

    // connect to name node    this.namenode = (DatanodeProtocol)       RPC.waitForProxy(DatanodeProtocol.class,                       DatanodeProtocol.versionID,                       nameNodeAddr,                        conf);

Tracing in the next rpc class:

VersionedProtocol proxy =        (VersionedProtocol) Proxy.newProxyInstance(            protocol.getClassLoader(), new Class[] { protocol },            new Invoker(addr, ticket, conf, factory));

Now, I understand!
1) the assignment of namenode is not a new object that implements the datanodeprotocol interface, but a dynamic proxy !!
2) in the above Code, the protocol type is datanodeprotocol. Class.
3) all calls to namenode are delegated to the invoker
Now, I understand!
1) the assignment of namenode is not a new object that implements the datanodeprotocol interface, but a dynamic proxy !!
2) in the above Code, the protocol type is datanodeprotocol. Class.
3) all calls to namenode are delegated to the invoker

4. Details of Heartbeat Layer 2: Look at the invoker class
The invoker class is a static internal class of the org. Apache. hadoop. IPC. RPC class:

 private static class Invoker implements InvocationHandler {

 

All the call methods are Delegate to the client!

public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {...      ObjectWritable value = (ObjectWritable)        client.call(new Invocation(method, args), address,                     method.getDeclaringClass(), ticket);      ...      return value.get();   }

Client is a member variable in invoker:

 private Client client;

Therefore, we can see that each method call in datanodeprotocol is encapsulated into an invocation object, which is then called by client. Call ().

5. Details of Heartbeat Layer 3: Invocation class

The invocation class is a static internal class of the org. Apache. hadoop. IPC. RPC class.

There is no business logic method, and its main role is a vo.

6. Details of Heartbeat Layer 4: client-class call Method

Next we will focus on the call method of the client class:

Public writable call (writable Param, inetsocketaddress ADDR, class <?> Protocol, usergroupinformation ticket) throws interruptedexception, ioexception {call = new call (PARAM); // convert invocation to call connection = getconnection (ADDR, protocol, ticket, call ); // connect to the remote server connection. sendparam (CALL); // send the parameter // send the "serialized" call to the previous Boolean interrupted = false; synchronized (CALL) {While (! Call. done) {try {call. wait (); // wait for the result // wait for the call result} catch (interruptedexception IE) {// Save the fact that we were interrupted = true ;}} if (interrupted) {// set the interrupt flag now that we are done waiting thread. currentthread (). interrupt ();} If (call. error! = NULL) {If (call. error instanceof RemoteException) {call. error. fillinstacktrace (); throw call. error;} else {// local exception throw wrapexception (ADDR, call. error) ;}} else {return call. value; // return }}}

 

7. Now, it's clear

The heartbeat Process sent by datanode to namenode is as follows: a) obtain the proxyb of namenode during datanode initialization) on datanode, call the heartbeat method of namenode Proxy: namenode. sendheartbeat (dnregistration, Data. getcapacity (), Data. getdfsused (), Data. getremaining (), xmitsinprogress. get (), getxceivercount (); c) the namenode dynamic proxy class on datanode wraps this call into (or serialized to) an invocation object and calls the client. call method D) The client call method converts invocation to the call Object E) the client sends the call to the real namenode server f) Nam After receiving the Enode, it is converted to the call at the namenode end, and after process, it is sent back through responder! G) datanode receives the result and converts the result to datanodecommand []

 

8. Check the dynamic proxy again.

Dynamic Proxy: it is possible to make "only interface, nothing corresponding implementation class", because the implementation of specific methods can be delegated to another class !!

In this example, for datanode, The datanodeprotocol interface does not implement classes!