Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode,hive.exec.parallel

來源:互聯網
上載者:User

Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode,hive.exec.parallel

動態分區數太大的問題:[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.


hive> insert into table sogouq_test partition(query_time) select user_id,query_word,query_order,click_order,url,query_time from sogouq_test_tmp;
Total MapReduce jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1409113942738_0026, Tracking URL = http://centos1:8088/proxy/application_1409113942738_0026/
Kill Command = /home/hadoop-2.2/bin/hadoop job  -kill job_1409113942738_0026
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-08-27 03:55:16,868 Stage-1 map = 0%,  reduce = 0%
[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.
Sample of 100 partitions created under hdfs://centos1:8020/hive/scratchdir/hive_2014-08-27_03-55-09_118_348369539322185503-1/_tmp.-ext-10002:
        .../query_time=20111230000005
        .../query_time=20111230000007
        .../query_time=20111230000008
        .../query_time=20111230000009
        .../query_time=20111230000010
        .../query_time=20111230000011


查看最大分區數:

hive> set hive.exec.max.dynamic.partitions.pernode;
hive.exec.max.dynamic.partitions.pernode=100

將該參數設定的大一點,問題即解決。


sata硬碟劃第16個分區時提示:The maximum number of partitions has been created,問怎突破限制?

邏輯分區理論上應該是23個,但是實踐中分區越少越好
用硬碟分區魔術師.最多能分26個區.在WINDOWS下不改變硬碟的內容都能夠繼續分多幾個區出來

才500g要分那麼多區?
 

相關文章

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.