Hadoop pipeline detailed [excerpt]

recently using a framework within the company to write map reduce found no encapsulation Hadoop streaming these things, check the next pipeline related thingsHadoop Pipeline Detailed

• .
• /
• Hadoop
• /
• 1 Comment

I. Description
Hadoop 2. X compared to 1.x has a large change, like the MapReduce layer architecture and the code is basically completely rewritten, in the HDFS layer added features such as ha,federation, code more hierarchical and easy to read, while the addition of PB in the early stage may be a barrier to reading, familiar with no big problem.
Pipeline one is an important process for Hadoop, whether it is an Mr Task, a hive task, and so on, and finally requires pipeline to write data to HDFs. So it makes sense to be familiar with the Hadoop pipeline process.

Ii. description of the process
The creation of Hadoop pipeline can be explained by the following flowchart, pipeline to establish namenode,datanode,dfsclient and other structures involved.

The following is a brief description
1, first dfsclient sends the RPC create request to Namenode, Namenode generates a new inodefile in the Inode and joins the new Leasemanager as the lease management to be written later.
2, after the successful return of the request, Dfsclient generates a outputstream, both Dfsoutputstream
3, the trip pipeline before sending a addblock request to Namenode, Namenode generate a new block, select the Datanode node to write, and register the block in Inodefile, and Blockmanager, Prepare to write
4, establish pipeline, according to the information returned by Namenode, find a primary datanode as the first node, ready to write
5, after the DataNode socket is received, the data is first written to buffer, then to the next node, after the successful writing, the data in the buffer is written to the local disk and waits for the ACK information
6, same as 5, write to the next node and write to the local, and finally wait for the ACK message
7, if the ACK is returned successfully, send back to Dfsclient, this write succeeds.

Third, detailed description

First, clarify some basic concepts of dfsclient writing Datanode. In large terms each file consists of one or more blocks, the block can have two meanings, one is the block in the Namenode,datanode, it is composed of block id,generate stamp. The other is the block stored in the Datanode disk, which really occupies the storage space.

The size of the block is typically defined in conf, which is 64M by default.

The Hadoop write process is transmitted through packet and chunk, each chunk is composed of 512Byte data bits and 4Byte parity bits, each packet is composed of 127 chunk, the size is 127* (512+4) = 65532Byte. Finally, if the write writes to the block size's boundary position, send a blank packet.

After confirming some of the above concepts, you can look at the pipeline. or in accordance with the two steps of the flowchart step by step analysis.
1. dfsclient send create request to Namenode
Dfsclients first sends a CREATE request to the Namenode, and Namenode first generates a new inodefile:

Inodefile newNode = Dir.addfile (src, permissions, replication, BlockSize, holder, Clientmachine, clientnode); if (NewNode = = null) { throw new IOException ("Unable to add" + src + "to namespace"); } Leasemanager.addlease (Newnode.getfileunderconstructionfeature () . Getclientname (), SRC);

And the lease,lease is added to the Leasemanager for lease management during the write process. The dfsclient is then added to the thread of a lease renewal, and the lease is periodically renewed to Namenode.

Beginfilelease (src, result);

2. Generate Dfsoutputstream

Final Dfsoutputstream out = new Dfsoutputstream (dfsclient, SRC, stat, flag, progress, checksum, favorednodes); Out.start ();

3, apply for a new block
When the PIPELINE is established, the state is Blockconstructionstage.pipeline_setup_create, and a new block is applied to the Namenode. The main code is:

Allocate new block, record block locations in INode. Newblock = Createnewblock (); Saveallocatedblock (SRC, Inodesinpath, newblock, targets); dir.persistnewblock (SRC, pendingfile); offset = pendingfile.computefilesize ();

The new block is associated with the Inodefile and is added to the Blockmanager.

4, establish pipeline
A socket connection is established with the first Datanode, the port is the IPC port, and the pipeline is ready to be established.

Send the request new Sender (out). WriteBlock (block, Accesstoken, Dfsclient.clientname, nodes, NULL, Recoveryflag? Stage.getrecoverystage (): Stage, Nodes.length, Block.getnumbytes (), BytesSent, Newgs, checksum, Cachingstrategy.get ()); //Receive ACK for connect Blockopresponseproto resp = blockopresponseproto.parsefrom ( pbhelper.vintprefixed (Blockreplystream));

Datanode starts the IPC Server, which has a dataxceiver dedicated to the IPC port, forwarding various requests, the code is:

/** Process op by the corresponding method. */Protected Final void Processop (Op op) throws ioexception {switch (OP) {case Read_block:opreadblock (); Break Case Write_block:opwriteblock (in); Break Case Replace_block:opreplaceblock (in); Break Case Copy_block:opcopyblock (in); Break Case Block_checksum:opblockchecksum (in); Break Case Transfer_block:optransferblock (in); Break Case Request_short_circuit_fds:oprequestshortcircuitfds (in); Break Case Release_short_circuit_fds:opreleaseshortcircuitfds (in); Break Case REQUEST_SHORT_CIRCUIT_SHM:OPREQUESTSHORTCIRCUITSHM (in); Break Default:throw new IOException ("Unknown op" + op + "in data stream"); } }

The written header its op is Write_block, which is handled by the WriteBlock method. When the stage is in the blockconstruncionstage.pipeline_setup_create phase when the PIPELINE is established, Datanode sends the requested data to the next node, and waits for the ACK information to be returned to the client with the code:

Read Connect ACK (only for clients, not FOR replication req) if (isclient) { Blockopresponseproto connectack =< C4/>blockopresponseproto.parsefrom (pbhelper.vintprefixed (Mirrorin)); Mirrorinstatus = Connectack.getstatus (); Firstbadlink = Connectack.getfirstbadlink (); if (log.isdebugenabled () | | Mirrorinstatus! = SUCCESS) { log.info ("Datanode" + targets.length + "got response fo R Connect Ack "+ " from downstream Datanode with Firstbadlink as "+ Firstbadlink); } }

At this point, pipeline is established, this stage does not involve the transmission of data, just try to set up pipeline and exception handling and so on.

5. Data transmission phase
The user gets the OutputStream first write to the local buffer, after writing a packet sent to primary DataNode, this time the stage in Blockconstruncionstage.data_ Streaming, the code is:

Get packet to be sent. if (Dataqueue.isempty ()) {One = new Packet (); Heartbeat Packet } else {One = Dataqueue.getfirst ();//Regular data Packet }

In the Datanode section, the data is received through the Blockreceiver code:

Receive the block and mirror to the next target if (blockreceiver! = null) { String mirroraddr = (Mirrorsock = = NULL)? Null:mirrornode; Blockreceiver.receiveblock (Mirrorout, Mirrorin, Replyout, mirroraddr, NULL, targets); Send Close-ack for transfer-rbw/finalized if (istransfer) { if (log.istraceenabled ()) { Log.trace (" Transfer:send close-ack "); } Writeresponse (SUCCESS, NULL, replyout); } }

Datanode first in the data received in buffer, the main code can see Blockreceiver Receivepacket method, first mirror to the next node:

First write the packet to the Mirror: if (mirrorout! = null &&!mirrorerror) { try { packetreceiver. Mirrorpacketto (mirrorout); Mirrorout.flush (); } catch (IOException e) { handlemirrorouterror (e); } }

Then write to Local:

Write data to disk. Out.write (Databuf.array (), Startbytetodisk, Numbytestodisk);

Write to the local time first in the TMP directory to establish two files, one is a data file, a checksum file (suffix meta), write the first time to write the data file, then update checksum, write to the checksum file.

6, after finishing the processing
If a block is finished, Dfsclient will close the pipeline with the following code:

private void Endblock () { if (DFSClient.LOG.isDebugEnabled ()) { DFSClient.LOG.debug ("Closing old block" + block ); } This.setname ("Datastreamer for file" + src); Closeresponder (); Closestream (); Setpipeline (null, NULL); stage = Blockconstructionstage.pipeline_setup_create; }

Then repeat the fourth step, if all finished, Datanode will report to Namenode receive block,dfsclient will report to Namenode complete, if the number of blocks did not reach the minimum number of copies, Complete needs to wait for a certain time to report, so pipeline completed.

Original Address http://dj1211.com/?p=178

Hadoop pipeline detailed [excerpt]