Hadoop Source Code Analysis: HDFs read and write Data flow control (Datatransferthrottler category)

Datatransferthrottler class Datanode control the transfer data rate when reading and writing data. This class is thread-safe and can be shared by multiple threads.

The purpose is to build the Datatransferthrottler object, and set the age period and bandwidth bandwidthpersec to call the Datatransferthrottler.throttle () method before the inter-read and write. Assuming that the I/O rate is too fast relative to the given bandwidth, the method will wait for the current thread.

Two constructors

• A dual-participation constructor that can set the period period and bandwidth bandwidthpersec.

Public Datatransferthrottler ( long period,long bandwidthpersec)

• The single-step constructor, the ability to set the bandwidth bandwidthpersec, the period period by default is set to 500ms.

Public Datatransferthrottler ( long bandwidthpersec)

Important attributes

Period	Period, per millisecond
Periodextension	Cycle expansion time. Unit milliseconds
Bytesperperiod	Total number of bytes that can be sent/received in a cycle
Curperiodstart	Current cycle start time, per millisecond
Curreserve	The number of bytes that can be sent/received in the current cycle
Bytesalreadyused	Number of bytes already in use in the current cycle

Bandwidth bandwidthperpersec = Bytesperperiod * period/1000

Datatransferthrottler.throttle () method

The method has two participation:

• Numofbytes is the number of bytes that need to be sent/received since the last time the throttle was called to today;
• The Canceler is an external cancellation device that can be used to detect if the flow control is canceled.

The Datatransferthrottler.throttle () method will first subtract the Curreserve from the numofbytes. Then run for example the following logic.

• Assuming that Curreserve is greater than 0, the current period is still surplus, and the throttle method returns directly.
• Assuming that Curreserve is less than or equal to 0, the current period has no margin and the following logic is run.

> Assuming that the current time now is less than the current period end time Curperiodend, wait to the next cycle

> Assume that the current time now is greater than the current period end time curperiodend and less than curperiodstart+periodextension. Description has entered the next cycle and throttle should not be used for a very long time, then curreserve plus the total number of bytes that can be transmitted in the next cycle bytesperperiod, and the Curperiodstart set to the start of the next cycle.

> Assume that the current time now is greater than curperiodstart+periodextension. The throttler may not be used for a very long time. The previous cycle is discarded.

The Datatransferthrottler.throttle () method source code such as the following:

 PublicsynchronizedvoidThrottle (LongNumofbytes, Canceler Canceler) {if(numofbytes <= 0) {return; }//Current period margin minus the number of bytes that need to be sent/received NumofbytesCurreserve-= numofbytes; bytesalreadyused + = Numofbytes;//Assuming Curreserve is less than or equal to 0, the current period has no margin while(Curreserve <= 0) {//Assuming a valid cancellation Canceler is passed in, and the cancellation state of the cancellation iscancelled is true, exit the while loop directlyif(Canceler! = null && canceler.iscancelled ()) {return; }Longnow = Monotonicnow ();//Calculates the current cycle end time. and stored in the curperiodend variable.LongCurperiodend = Curperiodstart + period;if(Now < Curperiodend) {//wait for the next cycle so that Curreserve can addTry{Wait (curperiodend-now); }Catch(Interruptedexception e) {//Terminate throttle, and reset the interrupted state to ensure that other interrupt processors in the call stack function correctlyThread. CurrentThread (). interrupt (); Break; }    }//Assuming the current time now is curperiodend late than the current end time and is less than curperiodstart+periodextension (3 times times the period), then go to the next cycle//and add Bytesperperiod to CurreserveElseif(Now < (Curperiodstart + periodextension))      {Curperiodstart = Curperiodend;    Curreserve + = Bytesperperiod; }//Assuming the current time now is greater than curperiodstart+periodextension, it may throttler for a very long time. Discard previous CycleElse{Curperiodstart = now;    Curreserve = bytesperperiod-bytesalreadyused; }} bytesalreadyused-= Numofbytes;}

Summarize

The Datatransferthrottler class is able to control the average rate of transmitted data over a period of time. However, the instantaneous rate may get out of control at the end of a cycle.

References

hadoop-release-2.5.0 Source Code

Reprint please attach original blog address: http://blog.csdn.net/jeff_fangji/article/details/44258827

Copyright notice: This article blog original articles, blogs, without consent, may not be reproduced.

Hadoop Source Code Analysis: HDFs read and write Data flow control (Datatransferthrottler category)