"Fix" hive dynamic increase partitions No more than 100 issues

Author:kwu

"Solve" hive dynamic increase partitions can not exceed 100 of the problem, the full dynamic generation partitions more than 100 will appear the following exception:

The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and Hive.exec.max.dynamic.part Itions.pernode. Maximum was set to:100

To resolve the 100 limit, you can set the following parameters:

Set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; Insert Overwrite table ods.cms_entity PARTITION (day) Select entity_id  , Entity_name  , Entity_desc  , Entity_ TYPE  , Entity_pid  , Entity_time  , entity_priority  , Entity_status  , Entity_channel  , Entity_ EDITOR  , Entity_template  , Entity_url    , Entity_category  , Entity_param  , Entity_shortname  , Entity_subtype  , Entity_compdelay  , day

Sets the dynamic partitions to 100000, which inserts a few years by the day of partitions more than 1000, successfully executes the statement.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

"Fix" hive dynamic increase partitions No more than 100 issues