In-depth analysis of MapReduce Architecture Design and Implementation Principles-Reading Notes (6) HadoopRPC Overview

The network communication module is the most underlying module in a distributed system. It directly supports complex inter-process communication (IPC) in the upper-layer distributed environment ). Is the basis of all Distributed Systems

The network communication module is the most underlying module in a distributed system. It directly supports complex inter-process communication (IPC) in the upper-layer distributed environment ). Is the basis of all Distributed Systems

The network communication module is the most underlying module in a distributed system. It directly supports complex inter-process communication (IPC) in the upper-layer distributed environment ). Is the basis of all distributed systems. Remote Procedure RPC is a common distributed network communication protocol. It allows programs running on one computer to call subprograms of another computer, while hiding network communication details.
Hadoop RPC features
1. Transparency
When a user calls a subroutine on another computer in a computer program, the user does not feel the call across machines.
2. High Performance
Each hadoop system adopts the Master/Slave structure, where the Master is actually an RPC Server, which is responsible for processing Server requests sent by all slave in the cluster. To ensure the concurrency of the Master, the RPC Server should be a high-performance Server that can effectively process concurrent RPC requests from multiple clients.
3. controllability
JDK comes with an RPC framework RMI, but hadoop is not used because it is too heavy. Instead, it completes a lightweight, high-performance, and controllable RPC framework.

Hadoop RPC consists of four parts. Serialization layer, function call layer, network transmission layer, and server processing framework.
Serialization layer: the structured object is designed for byte streams for transmission over the network or writing to persistent storage. In the RPC framework, it is mainly used to convert user request parameters or responses into word throttling for cross-machine transmission.
Hadoop implements its own serialization framework. As long as a class implements the Writable interface, it can support object serialization and deserialization.
Function call layer: locate the function to be called and execute the function. Hadoop RPC uses a java reflection mechanism and a dynamic proxy to call functions.
Network Transmission Layer: message transmission between the Client and the Server. Hadoop RPC uses a TCP/IP-based Socket mechanism.
Server Processing Framework: network I/O model. Describes how the client interacts with the server. Its design directly determines the server's concurrent processing capability. Hadoop RPC adopts the event-driven I/O model based on the Reactor design mode.

Hadoop RPC is divided into two layers.
The first layer is a Client/Server Communication Model Based on Java NIO.
The client encapsulates User call methods and parameters into a request package and sends them to the server. After receiving the request packet, the server returns the result to the server after a series of operations, such as unpacking, calling functions, and packaging results.
The second layer is the RPC interface directly called by upper-layer programs. These interfaces are the Client/Server Communication Models at the underlying layer.

Original article address: in-depth analysis of MapReduce Architecture Design and Implementation Principles-Reading Notes (6) HadoopRPC overview, thanks to the original author for sharing.