Analysis of communication protocol between Hadoop client and Datanode _java

This paper mainly analyzes the process of the Hadoop client read and write block. As well as client and Datanode communication protocols, data flow formats and so on.

The Hadoop client communicates with the Namenode via the RPC protocol, but the client and the Datanode communicate without using RPC, but instead use the socket directly, where the protocol for reading and writing is different, this article analyzes the Hadoop 0.20.2 version of (  Version 0.19 is the same. The principle and communication protocol of the communication between client and Datanode. In addition, the communication protocol between client and Datanode is changed in 0.23 and later versions, and Protobuf is used as a serialization method.

Write Block

1. The client first creates the file from the Namenode request through Namenode.create, and then starts the Datastreamer thread

2. The client consists of three threads, and the main thread is responsible for reading the local data into memory and encapsulating it as a package object and placing it in the queue dataqueue.

3. The Datastreamer thread detects whether the queue Dataqueue has package, and if so, creates the Blockoutputstream object (one block is created once, and a block may include multiple package), When created, it communicates with the corresponding Datanode, sends the Data_transfer_header information and gets the return. The responseprocessor thread is then created to receive the return ACK acknowledgement of the Datanode and error handling.

4. Datastreamer Dataqueue from the package object, sent to Datanode. Then continue to cycle to determine if Dataqueue has data ....

The following illustration shows the process of the write block.

The following figure is the format of the message

Read Block

Implemented primarily in the Blockreader class.

When Newblockreader is initialized,

1. Create a new Socketoutputstream (socket, timeout) by passing in the parameter sock, and write communication information that is not the same as the header of the write block.

Write the header.

Out.writeshort (Datatransferprotocol. Data_transfer_version );

Out.write (Datatransferprotocol. Op_read_block );

Out.writelong (Blockid);

Out.writelong (Genstamp);

Out.writelong (Startoffset);

Out.writelong (len);

Text. writestring (out, ClientName);

Out.flush ();

2. Create input stream new Socketinputstream (socket, timeout)

3. Judge return message In.readshort ()!= Datatransferprotocol. op_status_success

4. Create Checksum:datachecksum checksum = Datachecksum.newdatachecksum (in) according to the input stream

5. Read the position of the first chunk: Long Firstchunkoffset = In.readlong ()

Note: 512 bytes for one chunk Compute checksum (4 bytes)

6. Next read the specific data in the Blockreader Read method: result = Readbuffer (buf, off, Reallen)

7. One chunk read

int packetlen = In.readint ();

Long Offsetinblock = In.readlong ();

Long seqno = In.readlong ();

Boolean lastpacketinblock = In.readboolean ();

int datalen = In.readint ();

Ioutils. readfully (In, Checksumbytes.array (), 0,

Checksumbytes.limit ());

Ioutils. readfully (In, buf, offset, chunklen);

8. Checksum verification after reading data; Fsinputchecker.verifysum (Chunkpos)