[Translation] Using hadoop IPC/RPC for distributed applications

In distributed applicationsProgramUse hadoop IPC/RPC

Address: http://www.supermind.org/blog/520/using-hadoop-ipcrpc-for-distributed-applications

From: http://gpcuster.cnblogs.com

Translation:

About hadoop IPC

Let's take a look at hadoop IPC, a subsystem of inter-process communication. Hadoop IPC is used to communicate with programs of another process.

Hadoop IPC
1. Unlike soap and XML-RPC, hadoop IPC uses dataoutputstream and datainputstream in Java for binary serialization.
2. It is a simple and low-consumption RPC mechanism.
3. It only supports unicast operations.

Why is RMI or Java. Io. serialization not used for hadoop IPC implementation? The reason is taken from what Doug has to say:

Why not use serialization when we start the hadoop project? It seems too cumbersome to perform exact read and write operations on objects.

The reason for not applying RMI is similar. Effective IPC is crucial for hadoop. I think we need to precisely control these things, such as connection, timeout, cache, and so on. RMI cannot meet these requirements.

Example Code

Now let's take a look at the hadoop IPC sample code to see how they actually work.

Generally, all unicast PRC calls contain a client and a server.

Create a server,

 
Configuration conf =NewConfiguration ();
 

 
Server = rpc. getserver (This,"Localhost", 16000, conf );// Start a server on localhost: 16000
 

 
Server. Start ();
 

Create a client,

 
Configuration conf =NewConfiguration ();
 

 
Inetsocketaddress ADDR =NewInetsocketaddress ("Localhost", 16000 );// The server's inetsocketaddress
 

Clientprotocol client = (clientprotocol) RPC. waitforproxy (clientprotocol.Class,
 

 
Clientprotocol. versionid, ADDR, conf );
 

In this example, the class on the server implements the clientprotocol interface. The clientprotocol. Java code looks like this:

InterfaceClientprotocol extends org. Apache. hadoop. IPC. versionedprotocol {
 

 
Public StaticFinalLongVersionid = 1l;
 

 
Heartbeatresponse HEARTBEAT ();
 

 
}
 

The clientprotocol interface defines only one method: Heartbeat (), which returns a heartbeatresponse object. The remote client periodically calls the heartbeat () method to let the server know about the client. Then the server returns a heartbeatresponse object, and the client obtains the corresponding information.

A heartbeatresponse. Java code looks like this:

Public ClassHeartbeatresponse implements org. Apache. hadoop. Io. writable {
 

 
String status;
 

 
Public VoidWrite (dataoutputOut) Throws ioexception {
 

 
Utf8.writestring (Out, Status );
 

 
}
 

 
Public VoidReadfields (datainputIn) Throws ioexception {
 

 
This. Status = utf8.readstring (In);
 

 
}
 

 
}
 

Summary

The following is a summary of hadoop IPC:
1. The server implements the clientprotocol interface.

2. One or more clients will call the method of the clientprotocol interface.

3. All parameters or objects used in the clientprotocol interface method must inherit from org. Apache. hadoop. Io.