Hadoop RPC Protocol description--Go

Original address: https://spotify.github.io/snakebite/hadoop_rpc.html

Snakebite currently implements the following protocol in Snakebite.channel.SocketRpcChannel to communicate with The NameNode.

Connection

The Hadoop RPC protocol works as described below. On connection, headers is sent to setup a session. After this, multiple requests can be sent within the session.

Function	Type	Default
Header	bytes	"HRPC"
Version	Uint8	7
Auth method	Uint8	(Auth method Simple )
Serialization type	Uint8	0 (protobuf)
Ipcconnectioncontextproto length	UInt32
Ipcconnectioncontextproto	bytes

Sending messages

When sending a message, the following are sent to the sever:

TR class= "row-odd" >

Function	Type
Length of the next, parts	uint32
Rpcpayloadheaderproto length< /td>	varint
rpcpayloadheaderproto	Protobuf serialized message
hadooprpcrequestproto length	varint
hadooprpcrequestproto	protobuf serialized  Message

Varint is a Protocol Buffer variable int.

Note

The Java PROTOBUF implementation uses writetodelimited to prepend the message with their lenght, but the python I Mplementation doesn ' t implement such a method (yet).

Next to an rpckind   ( snakebites default is rpc_protocol_buffer ), an rpcop   ( snakebites default isrpc_final_payload ), the rpcpayloadheaderproto  message defines a callid  that is added in the RPC response (described below).

The Hadooprpcrequestproto contains a methodName field that defines what server method is called and a ha S a property request This contains the serialized actual request message.

Receiving messages

After a message was sent, the response can be read in the following-the-type:

function	type
Length of the Rpcresponseheaderproto	varint
Rpcresponseheaderproto	bytes
Length of the RPC response	uint32
serialized RPC response	bytes

The Rpcresponseheaderproto contains the callid of the request and a status field. The status can beSUCCESS, ERROR or FAILURE. In case SUCCESS The rest of response are a complete protobuf response.

In case of ERROR, the response looks like follows:

Function	Type
Length of the Rpcresponseheaderproto	Varint
Rpcresponseheaderproto	bytes
Length of the RPC response	UInt32
Length of the Exeption class name	UInt32
Exception class Name	Utf-8 String
Length of the stack trace	UInt32
Stack Trace	Utf-8 String

 

Hadoop RPC Protocol description--Go