Hadoop Source Code Analysis (40)

with the foundation above, we can dissect it. Dfsoutputstream up. First look at the constructor:

    PrivateDfsoutputstream (String src,LongBlockSize, progressable progress,        intbytesperchecksum)throws IOExceptionDfsoutputstream (String src, fspermissionmasked,BooleanOverwrite,         Shortreplication,Longblocksize,progressable Progress,        intbuffersize,intbytesperchecksum)throwsIOExceptionDfsoutputstream (String src,intbuffersize, progressable progress,Locatedblock Lastblock, Filestatusstat,        intbytesperchecksum)throwsIOException {

The parameters of these constructors are: file name src; Progress callback function Progress (reserved interface, not currently used); Chunk size Blocksize;block number of copies replication ; The size of each check chunk bytesperchecksum; file permission masked; Overwrite the original file tag overwrite; file status information stat; file's last block information lastblock;buffersize (? No references are seen). The
following two constructors call the first constructor, which invokes the constructor of the parent class and sets the Src,blocksize,progress and checksum properties of the object. The
second constructor calls the Namenode.create method, establishes the file in the file space, and starts Datastreamer, which is called by the dfsclient create method. The third constructor is called by the Dfsclient append method, and obviously this is a complex case, where the file has some data blocks and the added data is often added to the last block of data. At the same time, when the Append method is called, the client already knows some information about the last block's information and files, such as the block size, file permission bit, etc. contained in Filestatus. In conjunction with this information, the constructor needs to evaluate and set the values of some object member variables and attempt to recover from possible errors (call Processdatanodeerror) and finally start Datastreamer.
We look at the normal process, the previous analysis, through the FSOUTPUTSUMMER,HDFS client can transfer to the package, this packet is sent through the Writechunk, the following is their call relationship.

<ignore_js_op> 

After checking a series of states, Writechunk waits until the packets that are not sent in Dataqueue are less than the threshold value. If no packet object is available now, create a packet object to write data to packet, including checksum values and data. If the packet is written full, put it in the Send queue dataqueue. Writechunk the process is relatively simple, where the write, but also just write the data to the local queue, waiting for Datastreamer to send, not actually written to Datanode. The
Createblockoutputstream is used to establish a connection to the first Datanode, which is declared as follows:

Private Booleancreateblockoutputstream (datanodeinfo[] nodes, String client,                    BooleanRecoveryflag)

Nodes is a list of all Datanode that receive data, client is the name of the clients, and Recoveryflag indicates whether the connection was established for error recovery. Createblockoutputstream is simple, open the connection to the first Datanode, then send the packet in the format below, and wait for the ACK from Datanode. If an error occurs, record the location of the Datanode in nodes, set the ErrorIndex, and return false.
<ignore_js_op>

When the Recoveryflag indication is true, it means that this write is a recovery operation, and for Datanode, this means that the temporary files prepared for writing (in the TMP directory) may already exist and require some special handling, see the implementation of Fsdataset.
The Nextblockoutputstream method can be called when the client writes data that requires a new block.

    Privatedatanodeinfo[] Nextblockoutputstream (String client)throwsIOException

The implementation of this method is simple, first calling Locatefollowingblock (contains retry and error handling), Get a new block of data by Namenode.addblock, return a list of datanodeinfo, and with this list, you can create a pipe that writes data. The next big thing is to call the createblockoutputstream above and establish a connection to the Datanode.
With the preparation above, we analyze Processdatanodeerror, whose main process is:

l          parameter checking;l          Turn off Blockstream and blockreplystream;l  that may also be open         move data blocks that are not answered (in Ackqueue) to Dataqueue ;l          loop execution:1.       Calculate the Datanode list that is still alive;2.     & nbsp; Select a primary Datanode, start the recovery process above it by Datanode RPC Recoverblock method;3.       handling possible errors;4.        handles possible changes to the block after recovery (such as stamp changes);5.       calls Createblockoutputstream to Datanode connections. l          start responseprocessor.

This process involves the Recoverblock method on the Datanode and the possible block recovery in the Createblockoutputstream, which is a fairly resource-intensive approach, when the probability of a system error is relatively small, And there's a lot of data that can be recovered on the data block (average 32M), it's worth it.
The process of writing is analyzed, and then we look at the closure of the stream, which also involves a series of methods whose invocation relationships are as follows:

<ignore_js_op>


Flushinternal waits until the send queue (including the possible currentpacket) and the reply queue are empty, which means that the data is Datanode successfully received.
Sync acts like a Unix sync, and writes data is persisted. It first calls the parent class's Flushbuffer method, copies the data that might not have been copied to the Dfsoutputstream, and then calls Flushinternal, waiting for all the data to be written. Then call Namenode.fsync to persist the data on the namespace.
Closeinternal is a bit more complicated, it first calls the parent class's Flushbuffer method, copies the data that might not have been copied to Dfsoutputstream, and then calls Flushinternal, waiting for all the data to be finished. Then end the two worker threads, close the socket, and finally call Amenode.complete, notifying Namenode to end the write operation. The Close method calls Closeinternal first, and then removes the corresponding information from the local leasechecker.

For more highlights, please follow: http://bbs.superwu.cn

Focus on the two-dimensional code of Superman Academy:

Hadoop Source Code Analysis (40)