Hive count distinct, hivedistinct

Hive count distinct, hivedistinct

Select count (distinct user_id) from dm_user where ds = 20150701;

With the disticnt function, all data is shuffled to only one CER, causing serious data skew of reducer.

After optimization

Set mapred. reduce. tasks = 50;

Select count (*) from

(Select user_id from Maid where ds = 20150701 group by user_id) t;

Order by global sorting, only one reduce

Sort by in a reduce, and distripartition by field is divided into different reduce

Distribute by is first divided into different reduce sorting, and then sorted within reduce.

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.