006, Hadoop-rpc the bottom of the implementation of the detailed

the first part: what is RPC

RPC (remote Procedure call Protocol) – Remoting procedure protocol invocation. With RPC, we can request services from computers on the network without needing to know the underlying network protocols. The interactions at the bottom of Hadoop are all made through RPC. For example, communication between Datanode and Namenode, Tasktracker and Jobtracker, secondary namenode, and Namenode is implemented via RPC.

RPC Mode

RPC takes client/server mode . The requestor is a client, and the service provider is a server. First, the client call process sends a call message with process parameters to the service process, and then waits for the reply message. On the server side, the process stays asleep until the call information arrives. When a call arrives, the server obtains the process parameters, evaluates the result, sends a reply message, and then waits for the next invocation information, and finally, the client invokes the process to receive the reply message, obtains the process result, and then invokes execution to proceed.

Working principle

At runtime, a client-to-server RPC call, with an internal operation of roughly 10 steps, is as follows:

1. Invoking the client handle; executing the transfer parameters

2. Call the local system kernel to send a network message

3. Messaging to a remote host

4. Server handle get message and get parameters

5. Performing a remote procedure

6. The execution process returns the result to the server handle

7. Server handle returns result, call remote system kernel

8. Send the message back to the local host

9. Client handles receive messages from the kernel

10. The customer receives the data returned by the handle

Part II: Hadoop of the RPC mechanism

Hadoop PRC

1.Hadoop RPC is widely used throughout Hadoop, and communication between the Client, DataNode, and Namenode depends on it.

For example, when we operate HDFS normally, we use the FileSystem class, which has a Dfsclient object inside it, which is responsible for dealing with Namenode. At run time, Dfsclient creates a Namenode proxy locally, and then it operates on the proxy, and the proxy is remotely called to the Namenode method via the network, and can also return a value.

2.Hadoop RPC in ORG.APACHE.HADOOP.IPC

3.Hadoop RPC = Dynamic Proxy + custom-made binary stream

4. Divided into server and clinet side

Service-side process

The 1.Listener thread monitors the data that RPC client sends over.

2. When there is data to be received, call connection's Readandprocess method.

3. The connection side receives the data processing, if receives a complete call package, constructs a call object. Push to the call queue, the handler thread handles all call in the call queue.

The 4.Handler thread listens to the call queue and, if the call queue is not empty, removes call from the call queue by a FIFO rule.

5. Give call to rpc.server for processing.

6. Use the method provided by the JDK to complete the call to the target method, which is implemented by the specific business logic.

7. Return the response. Server.handler sends a response to the RPC client in an asynchronous, non-blocking manner, and if there is data that is not sent, it is done by Server.responder.

006, Hadoop-rpc the bottom of the implementation of the detailed