hive count distinct,hivedistinct

來源:互聯網
上載者:User

hive count distinct,hivedistinct

select count(distinct user_id) from dm_user where ds=20150701;

使用disticnt函數,所有的資料只會shuffle到一個reducer上,導致reducer資料扭曲嚴重

最佳化後為

set mapred.reduce.tasks=50;

select count(*) from

(select user_id from dm_user where ds=20150701 group by user_id)t;


order by全域排序,只有一個reduce

sort by 在一個reduce中排序,distribute by 按欄位分為不同的reduce    

distribute by 先分為不同的reduce排序,之後在reduce內部排序

著作權聲明:本文為博主原創文章,未經博主允許不得轉載。

相關文章

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.