hive中的distribute By

hive中的distribute by是控制在map端如何拆分資料給reduce端的。
hive會根據distribute by後面列，根據reduce的個數進行資料分發，預設是採用hash演算法。

對於distribute by進行測試，一定要分配多reduce進行處理，否則無法看到distribute by的效果。

hive> select * from test09;
OK
100 tom
200 mary
300 kate
400 tim
Time taken: 0.061 seconds

hive> insert overwrite local directory ‘/home/hjl/sunwg/ooo’ select * from test09 distribute by id;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201105020924_0070, Tracking URL = http://hadoop00:50030/jobdetails.jsp?jobid=job_201105020924_0070
Kill Command = /home/hjl/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=hadoop00:9001 -kill job_201105020924_0070
2011-05-03 06:12:36,644 Stage-1 map = 0%, reduce = 0%
2011-05-03 06:12:37,656 Stage-1 map = 50%, reduce = 0%
2011-05-03 06:12:39,673 Stage-1 map = 100%, reduce = 0%
2011-05-03 06:12:44,713 Stage-1 map = 100%, reduce = 50%
2011-05-03 06:12:46,733 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201105020924_0070
Copying data to local directory /home/hjl/sunwg/ooo
Copying data to local directory /home/hjl/sunwg/ooo
4 Rows loaded to /home/hjl/sunwg/ooo
OK
Time taken: 17.663 seconds

第一次執行根據id欄位來做分發，結果如下：

[hjl@sunwg src]$ cat /home/hjl/sunwg/ooo/attempt_201105020924_0070_r_000000_0
400tim
200mary
[hjl@sunwg src]$ cat /home/hjl/sunwg/ooo/attempt_201105020924_0070_r_000001_0
300kate
100tom

這次我們換個分發的方式，採用length(id)的結果，因為這幾條記錄的id欄位的長度都相同，所以應該會被分布到同一個reduce中。

hive> insert overwrite local directory ‘/home/hjl/sunwg/lll’ select * from test09 distribute by length(id);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201105020924_0071, Tracking URL = http://hadoop00:50030/jobdetails.jsp?jobid=job_201105020924_0071
Kill Command = /home/hjl/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=hadoop00:9001 -kill job_201105020924_0071
2011-05-03 06:15:21,430 Stage-1 map = 0%, reduce = 0%
2011-05-03 06:15:24,454 Stage-1 map = 100%, reduce = 0%
2011-05-03 06:15:31,509 Stage-1 map = 100%, reduce = 50%
2011-05-03 06:15:34,539 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201105020924_0071
Copying data to local directory /home/hjl/sunwg/lll
Copying data to local directory /home/hjl/sunwg/lll
4 Rows loaded to /home/hjl/sunwg/lll
OK
Time taken: 20.632 seconds

在查看下結果是否和我們的預期相同：
[hjl@sunwg src]$ cat /home/hjl/sunwg/lll/attempt_201105020924_0071_r_000000_0
[hjl@sunwg src]$ cat /home/hjl/sunwg/lll/attempt_201105020924_0071_r_000001_0
100tom
200mary
300kate
400tim

檔案attempt_201105020924_0071_r_000000_0中沒有記錄，而全部的記錄都在attempt_201105020924_0071_r_000001_0中。

 

轉自http://www.oratea.net/?p=626