"Resolve" hive dynamic add partitions no more than 100 issues

Author:kwu

"Resolve" hive dynamic add partitions can not exceed 100 problem, full dynamic generation partitions more than 100 will occur such as the following exception:

The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and Hive.exec.max.dynamic.part Itions.pernode. Maximum was set to:100

To resolve the 100 limit, you can set the following parameters such as:

Set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; Insert Overwrite table ods.cms_entity PARTITION (day) Select entity_id, Entity_name, Entity_desc, Entity_type, entity_p ID, Entity_time, entity_priority, Entity_status, Entity_channel, Entity_editor, Entity_template, ENTITY_URL, ENT Ity_category, Entity_param, Entity_shortname, Entity_subtype, Entity_compdelay, Dayfrom stage.cms_entity_by_day;

Set the dynamic partitions to 100000, which inserts a few days partitions more than 1000, and runs the statement successfully.

"Resolve" hive dynamic add partitions no more than 100 issues