RPC and Hadoop RPC mechanism

First, what is RPC?

(1) remote Procdure call, which allows a computer program to remotely invoke the subroutine of another computer without worrying about the underlying network communication details, is transparent to us. Often used in distributed network communication.

(2) The process of Hadoop interaction is dead through RPC, such as Namenode and Datanode Direct, between Jobtracker and Tasktracker.

Process:

(1) RPC adopts C/s mode;

(2) The client sends a request message with parameters to the server;

(3) After receiving this request, the server calls the corresponding program according to the parameters sent, then sends its calculated results to the client side;

(4) The client side will continue to operate after receiving the result;

RPC Features

(1) Transparency: remote invocation of programs on other machines is like calling local methods to the user;

(2) High performance: RPC server can concurrently handle multiple requests from the client;

(3) Controllability: An RPC framework--rmi has been provided in the JDK, but the PRC framework is too heavyweight and less controllable, so Hadoop RPC implements a custom PRC framework.

Second, the RPC mechanism of Hadoop

(1) serialization layer: Clent and server-side communication transfer information using the serialized class provided in Hadoop or the custom writable type;

(2) Function call Layer: Hadoop RPC implements function call through dynamic Proxy and Java reflection;

(3) Network Transport layer: Hadoop RPC uses TCP/IP-based socket mechanism;

(4) server-side framework layer: RPC Server uses Java NIO and the event-driven I/O model to improve RPC server concurrency processing power;

Hadoop RPC Design Technology

(1) Dynamic agent

(2) Reflection--Dynamic loading class

(3) serialization

(4) Non-blocking asynchronous IO (NIO)

Java Dynamic Agent

Related classes and interfaces:

1.java.lang.reflect.proxy

2.java.lang.reflect.invocationhandler

Dynamic Proxy Creation Object procedure:

1.InvocationHandler handler = new Invocationhandlerimpl ();

2.proxy.newproxyinstance ();

(1) The dynamic agent can provide access to another object, while hiding the concrete facts of the actual object, and the proxy object hides the actual object from the customer.

(2) The dynamic agent can do some other things to the request, in the not allowing direct access to certain classes, or need to do some special processing of access, you can consider the use of proxies.

(3) Support for dynamic proxies is currently available in the Java development package, but only the implementation of interfaces is now supported.

RPC and Hadoop RPC mechanism