Hadoop RPC Remote Procedure Call source parsing and example

Introduction to RPC:

RPC remote Procedure call, which allows a computer program to remotely invoke the subroutine of another computer without worrying about the underlying network communication details, is transparent to us. Often used in distributed network communication.

Inter-process interaction between Hadoop is done through RPC, such as between Namenode and Datanode, between Jobtracker and Tasktracker.

The RPC protocol assumes that some transport protocols exist, such as TCP or UDP, to carry information data between communication programs. In the OSI network communication model, RPC spans the transport and application tiers. RPC makes it easier to develop applications that include distributed, multi-program networks.

RPC takes client/server mode. The requestor is a client, and the service provider is a server.

First, the client call process sends a call message with process parameters to the service process, and then waits for the reply message on the server side, and the process stays asleep until the call information arrives. When a call arrives, the server obtains the process parameters, evaluates the result, sends a reply message to the client, and then waits for the next call message, and finally, the client invokes the process to receive the reply message, obtains the process result, and then invokes execution to proceed.

RPC Features:

(1) Transparency: Remotely invoking a program on another machine is like calling a local method to a user.

(2) High performance: RPC server can concurrently process multiple requests from the client (request queue). 3. controllability: An RPC framework –rmi has been provided in the JDK, but the RPC framework is too heavyweight and controllable, so HADOOPRPC implements the custom RPC framework.

Hadoop RPC Communication:

(1) Serialization layer: The information that the client communicates with the server side uses the serialized class or custom writable type provided in Hadoop.

(2) Function call Layer: Hadoop RPC implements function calls through dynamic proxies and Java reflection mechanisms.

(3) Network Transport layer: Hadoop RPC uses a TCP/IP-based socket mechanism.

(4) server-side framework layer: RPC Server uses Java NIO and the event-driven I/O model to improve the concurrency of RPC server.

The entire architecture of Hadoop is built on top of RPC (ORG.APACHE.HADOOP.IPC).

Hadoop RPC Design Technology

(1) Dynamic agent

(2) Reflection 3, serialization 4, non-blocking asynchronous IO (NIO)

Dynamic Agent:

(1) The dynamic agent can provide access to another object, while hiding the concrete facts of the actual object, and the proxy object hides the actual object from the customer.

(2) The dynamic agent can do some other processing of the request, when you do not allow direct access to certain classes, or need to do some special processing of access, you can consider using a proxy. 3) support for dynamic proxies is currently available in the Java development package, but only the implementation of interfaces is now supported. Related classes and interfaces: java.lang.reflect.proxy--class java.lang.reflect.invocationhandler--interface

Dynamic Proxy Creation Object procedure:

Invocationhandler handler= new Invocationhandlerimpl (...) Proxy.newinstance (...)

The concrete implementation can refer to the following:

According to view hadoop2.6.0 source code

Client:

Server:

Rpc:

A few important agreements:

(1) ClientProtocol is the interface that the client (FileSystem) communicates with Namenode.

(2) Datanodeprotocol is the interface between Datanode and Namenode communication Namenodeprotocol is the interface of Secondarynamenode and Namenode communication.

(3) Dfsclient is an object that calls the Namenode interface directly. User code calls Dfsclient objects through Distributedfilesystem to deal with Namenode.

Simulating Hadoop RPC Traffic

Package myrpc;

Import Org.apache.hadoop.io.Text;

Import Org.apache.hadoop.ipc.VersionedProtocol;

Public interface Myrpcprotocal extends versionedprotocol{

public static Long VersionID = 23234l;//It's important to have an afternoon to solve it.

Public text test (text T);

}

Package myrpc;

Import java.io.IOException;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.io.Text;

Import Org.apache.hadoop.ipc.ProtocolSignature;

Import Org.apache.hadoop.ipc.RPC;

Import Org.apache.hadoop.ipc.RPC.Server;

public class Rpcserver implements myrpcprotocal{

Server server = NULL;

Public Rpcserver () throws IOException, interruptedexception{

Server = Rpc.getserver (this, "localhost", 8888,new Configuration ());

Slightly changed relative to previous versions

Rpc. Builder ins = new RPC. Builder (New Configuration ());

Ins.setinstance (this);

Ins.setbindaddress ("localhost");

Ins.setport (9999);

Ins.setprotocol (Myrpcprotocal.class);

Rpc.setprotocolengine (New Configuration (), Myrpcprotocal.class, Rpcengine.class);

Server = Ins.build ();//Get a server instance

Server.start ();

Server.join ();

}

public static void Main (string[] args) throws IOException, Interruptedexception {

New Rpcserver ();

}

@Override

Public long Getprotocolversion (String protocol, long ClientVersion)

Throws IOException {

return Myrpcprotocal.versionid;

}

@Override

Public Protocolsignature Getprotocolsignature (String protocol,

Long clientversion, int clientmethodshash) throws IOException {

return new Protocolsignature ();

}

@Override

Public text test (text t) {

if (t.tostring (). Equals ("RPC")) {

return new Text ("OK");

}

return new Text ("false");

}

}

Package myrpc;

Import java.net.InetSocketAddress;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.io.Text;

Import Org.apache.hadoop.ipc.RPC;

public class Rpcclient {

Private Myrpcprotocal protocal;

Public Rpcclient () throws exception{

Inetsocketaddress address = new Inetsocketaddress ("localhost", 9999);

Protocal = (myrpcprotocal) rpc.waitforproxy

(Myrpcprotocal.class,myrpcprotocal.versionid, Address, New Configuration ());

Rpc.setprotocolengine (New Configuration (), Myrpcprotocal.class, Rpcengine.class);

}

public void Call (String s) {

Final text string = protocal.test (new text (s));

System.out.println (String.tostring ());

}

public static void Main (string[] args) throws Exception {

Rpcclient client = new Rpcclient ();

Client.call ("RPC");

}

}

Hadoop RPC Remote Procedure Call source parsing and example