Hadoop RPC Remote Procedure Call source parsing and example

What is RPC?

1. RPC (remote Procedure Call) remoting procedure calls, which allow a computer program to remotely invoke the subroutine of another computer without having to care about the underlying network communication details , is transparent to us. Often used in distributed network communication.
2, the process of Hadoop interaction is through RPC, such as between Namenode and Datanode, between Jobtracker and Tasktracker.

The RPC protocol assumes that some transport protocols exist, such as TCP or UDP, to carry information data between communication programs. In the OSI network communication model, RPC spans the transport and application tiers. RPC makes it easier to develop applications that include distributed, multi-program networks.

RPC takes client/server mode. The requestor is a client, and the service provider is a server.
First, the client call process sends a call message with process parameters to the service process , and then waits for the reply message.
On the server side, the process stays asleep until the call information arrives . When a call arrives, the server obtains the process parameters, evaluates the results, sends a reply message to the client,
And then wait for the next call message ,
Finally, the client calls the process to receive the reply information, obtains the process result , and then invokes execution to proceed.

RPC Features

1. Transparency: Remotely invoking programs on other machines is like calling a local method to the user.
2, High performance: RPC server can concurrently handle multiple requests from the client (request queue).
3. controllability: An RPC framework –rmi has been provided in the JDK, but the RPC framework is too heavyweight and controllable, so Hadoop RPC implements a custom RPC framework.

Hadoop RPC Communication

1, serialization layer: The client and server-side communication to pass the information used in Hadoop provided by the serialization class or custom writable type.

2. Function call Layer: Hadoop RPC implements function calls through dynamic proxies and Java reflection mechanisms .

3, the network Transport layer: Hadoop RPC uses TCP/IP-based socket mechanism.

4. Server-side framework layer: RPC Server uses Java NIO and the event-driven I/O model to improve the concurrency of RPC server .

The entire architecture of Hadoop is built on top of RPC (ORG.APACHE.HADOOP.IPC).

Hadoop RPC Design Technology

1. Dynamic Agent
2. Reflection
3. Serialization
4. Non-blocking asynchronous IO (NIO)

Dynamic Agent

1. The dynamic agent can provide access to another object while hiding the concrete facts of the actual object, and the proxy object hides the actual object from the customer.
2, the dynamic agent can do some other processing of requests, in not allowing direct access to certain classes, or need to do some special processing of access, this time can consider the use of proxies.
3) support for dynamic proxies is currently available in the Java development package, but only the implementation of interfaces is now supported.
Related classes and interfaces:

java.lang.reflect.Proxy   --类java.lang.reflect.InvocationHandler  --接口

Dynamic Proxy Creation Object procedure:

InvocationHandler  handler = new InvocationHandlerImpl(...)Proxy.newInstance(...)

The concrete implementation can refer to the following:
http://blog.csdn.net/scgaliguodong123_/article/details/44025531

According to view hadoop2.6.0 source code

Client

Server

RPC

a few important agreements

ClientProtocol is the interface that the client (FileSystem) communicates with Namenode.
Datanodeprotocol is the interface between Datanode and Namenode communication
Namenodeprotocol is Secondarynamenode and Namenode.
The interface of the communication.
dfsclient is an object that calls the Namenode interface directly. User code calls Dfsclient objects through Distributedfilesystem to deal with Namenode.

simulating Hadoop RPC traffic

package MyRPC;import org.apache.hadoop.io.Text;import org.apache.hadoop.ipc.VersionedProtocol;publicinterface MyRPCProtocal extends VersionedProtocol{    publicstaticlong23234l;//很重要很重要，搞了一下午才解决掉。    publictest(Text t);}

 PackageMYRPC;ImportJava.io.IOException;ImportOrg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.ipc.ProtocolSignature;ImportOrg.apache.hadoop.ipc.RPC;ImportOrg.apache.hadoop.ipc.RPC.Server; Public  class rpcserver implements myrpcprotocal{Server Server =NULL; Public Rpcserver()throwsIOException, interruptedexception{//server = Rpc.getserver (this, "localhost", 8888,new Configuration ());        //Slightly altered relative to previous versionsRpc. Builder ins =NewRpc. Builder (NewConfiguration ()); Ins.setinstance ( This); Ins.setbindaddress ("localhost"); Ins.setport (9999); Ins.setprotocol (Myrpcprotocal.class);//rpc.setprotocolengine (New Configuration (), Myrpcprotocal.class, rpcengine.class);Server = Ins.build ();//Get a server instanceServer.start ();      Server.join (); } Public Static void Main(string[] args)throwsIOException, Interruptedexception {NewRpcserver (); }@Override     Public Long getprotocolversion(String protocol,LongClientVersion)throwsIOException {returnMyrpcprotocal.versionid; }@Override     PublicProtocolsignaturegetprotocolsignature(String protocol,LongClientVersion,intClientmethodshash)throwsIOException {return NewProtocolsignature (); }@Override     PublicTextTest(Text t) {if(T.tostring (). Equals ("RPC")){return NewText ("OK"); }return NewText ("false"); }}

 PackageMYRPC;Importjava.net.InetSocketAddress;ImportOrg.apache.hadoop.conf.Configuration;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.ipc.RPC; Public  class rpcclient {    PrivateMyrpcprotocal protocal; Public rpcclient()throwsexception{inetsocketaddress address =NewInetsocketaddress ("localhost",9999); Protocal = (myrpcprotocal) rpc.waitforproxy (Myrpcprotocal.class,myrpcprotocal.versionid, address,NewConfiguration ());//rpc.setprotocolengine (New Configuration (), Myrpcprotocal.class, rpcengine.class);} Public void Pager(String s) {FinalText string = Protocal.test (NewText (s));    System.out.println (String.tostring ()); } Public Static void Main(string[] args)throwsException {Rpcclient client =NewRpcclient (); Client.call ("RPC"); }}

Hadoop RPC Remote Procedure Call source parsing and example