Hadoop RPC mechanism

RPC (remote Procedure call Protocol) remoting protocol, a protocol that requests services from a remote computer program over a network without needing to know the underlying network technology. The interactions at the bottom of Hadoop are all made through RPC. For example, communication between Datanode and Namenode, Tasktracker and Jobtracker, secondary namenode, and Namenode is implemented via RPC. The following is a diagram of the RPC interaction process:

the entity model of the RPC server side
The upper part is the user's point of view, macroscopic observation of the entire call process. This section analyzes what entities RPC has in detail. Why should we mention these entities? If the RPC process as a pipeline, these entities are to do a specific job of workers, if you want to understand the pipeline processing, you need to know each job his responsibilities and profile.

RPC does not have much detail on the client, just to mention how RPC intercepts the call request when the user invokes the proxy object. A clear friend of the dynamic agent knows that when creating a proxy object, you need to associate it with a invocationhandler, each call to the proxy object goes into the bound Invocationhandler, and RPC gets the user's request from here, there is no doubt. [About dynamic agents]

What needs to be said in detail is the RPC on the server model, which consists of a series of entities that are responsible for invoking the entire process. You can also use a picture to describe them here.


From the figure, each entity has a clear division of labor and responsibilities. I'll show you the following.
Listener
Listens to the RPC server port, if the client has a connection request arrives, it accepts the connection, and then forwards the connection to a reader, allowing reader to read the connected data. If there is more than one reader, the order is distributed among the reader when a new connection comes in. It should be mentioned here that the Hadoop0.21 version has a bug (JIRA) when supporting multiple reader, and the server process does not shut down properly if reader is not used during server run time
Reader
Reader's job is to read the data stream from a client connection and then convert it to the calling object (call) and put it in the call queue.
Handler
The entity that really does things. It gets the call information from the call queue, then the reflection invokes the real object, gets the result, and then puts the call into the response Queue (response queue)
Responder
It constantly checks the response queue for any call information and, if so, returns the result of the call to the client.

The entire invocation process is related to the network and is treated with NIO.

Hadoop RPC mechanism