Jobtracker hang problems caused by hive dynamic partitioning

Familiar with the Jobtracker is known, in the job initialization eagerhttp://www.aliyun.com/zixun/aggregation/17034.html "> Taskinitializationlistener will lock the jobinprogress and then inittask, details please check the code, here is a step is to write the initial data to HDFs and flush, and Fairscheduler update thread, when updating resource pool resources, is holding exclusive locks on Jobtracker and Fairscheduler and then calculating resource conditions for each resource pool, while computing running_map/running_ Reduce time to get the corresponding jobinprogress lock, readers may not understand why I want to talk about this piece, the problem appears here.

Hive in processing dynamic partitions, the main experience of such a few steps tablescan->filesink->movetask

When the Filesink is performed according to the record, the N (part) writer is then started to process the dynamic partition field, that is, DT here, if DT is contiguous then open a block to start writing, otherwise close the current block and open the new dir Block continues to write, where if the DT is discontinuous and the record number is large, a large number of files are generated, resulting in a HDFS load elevation, which is matched to the HDFS monitoring:

Cluster load at that time:

Number of documents produced at that time:

Then lead to jobinprogress is locked, thus Jobtracker is locked, lead jobtracker hang Live!

So how do we solve it? using Distributeby DT to arrange the same dt together and then Filesink will not cause a large number of small files produced.