MapReduce效能分析實驗

標籤：

最近應項目需要, 對MapReduce進行了一些實驗測試, 記錄如下.

 

測試環境

3台VM虛擬機器, 都是Ubuntu系統, 1G記憶體, Hadoop 2.6.0

1台 NameNode (Master)

3台 DataNode (Slave)

其中Master和2台Slave (Slave2, Slave3) 位於一配置較強的物理機中, 另1Slave (Slave1) 位於一配置較差的物理機.

 

資料準備

共28個文字檔, 每個檔案大概12M, 共約330M的資料

 

其內容大致是

 

實驗1 節點任務分布情況

我們的測試程式就是基本的單詞計數程式.

package MyPackage;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;        public class WordCount {         public static class Map extends Mapper<Object, Text, Text, IntWritable> {    private final static IntWritable one = new IntWritable(1);    private Text word = new Text();            public void map(Object key, Text value, Context context) throws IOException, InterruptedException {        String line = value.toString();        StringTokenizer tokenizer = new StringTokenizer(line);        while (tokenizer.hasMoreTokens()) {            word.set(tokenizer.nextToken());            context.write(word, one);        }    } }   public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {    public void reduce(Text key, Iterable<IntWritable> values, Context context)       throws IOException, InterruptedException {        int sum = 0;        for (IntWritable val : values) {            sum += val.get();        }        context.write(key, new IntWritable(sum));    } }   public static void main(String[] args) throws Exception {    Configuration conf = new Configuration();                Job job = new Job(conf, "wordcount");            job.setJarByClass(WordCount.class);            job.setOutputKeyClass(Text.class);    job.setOutputValueClass(IntWritable.class);        job.setCombinerClass(Reduce.class);        job.setMapperClass(Map.class);    job.setReducerClass(Reduce.class);                FileInputFormat.addInputPath(job, new Path(args[0]));    FileOutputFormat.setOutputPath(job, new Path(args[1]));                job.waitForCompletion(true); }        }

 

實驗結果

總共花了約11分鐘完成任務

共啟動了28個Map Task

28個Map Task在3個Slave上並非平均分布.

Attempt	State	Status	Node	Logs	Start Time	Finish Time	Elapsed Time	Note
attempt_1430534608975_0001_m_000000_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:47:03 GMT	3mins, 0sec
attempt_1430534608975_0001_m_000001_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:08 GMT	3mins, 5sec
attempt_1430534608975_0001_m_000002_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:08 GMT	3mins, 5sec
attempt_1430534608975_0001_m_000003_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:07 GMT	3mins, 4sec
attempt_1430534608975_0001_m_000004_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:08 GMT	3mins, 5sec
attempt_1430534608975_0001_m_000005_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:12 GMT	3mins, 9sec
attempt_1430534608975_0001_m_000006_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:08 GMT	3mins, 5sec
attempt_1430534608975_0001_m_000007_0	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:44:03 GMT	Sat, 02 May 2015 02:47:00 GMT	2mins, 57sec
attempt_1430534608975_0001_m_000008_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:31 GMT	1mins, 28sec
attempt_1430534608975_0001_m_000009_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:55 GMT	1mins, 52sec
attempt_1430534608975_0001_m_000010_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:59 GMT	1mins, 57sec
attempt_1430534608975_0001_m_000011_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:54 GMT	1mins, 52sec
attempt_1430534608975_0001_m_000012_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:31 GMT	1mins, 28sec
attempt_1430534608975_0001_m_000013_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:44:02 GMT	Sat, 02 May 2015 02:45:55 GMT	1mins, 52sec
attempt_1430534608975_0001_m_000014_1	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:47:18 GMT	Sat, 02 May 2015 02:47:23 GMT	4sec
attempt_1430534608975_0001_m_000015_0	SUCCEEDED	map	/default-rack/Slave1:8042	logs	Sat, 02 May 2015 02:45:08 GMT	Sat, 02 May 2015 02:50:42 GMT	5mins, 33sec
attempt_1430534608975_0001_m_000016_0	SUCCEEDED	map	/default-rack/Slave1:8042	logs	Sat, 02 May 2015 02:45:08 GMT	Sat, 02 May 2015 02:50:57 GMT	5mins, 48sec
attempt_1430534608975_0001_m_000017_1	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:47:48 GMT	Sat, 02 May 2015 02:47:52 GMT	4sec
attempt_1430534608975_0001_m_000018_1	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:48:18 GMT	Sat, 02 May 2015 02:48:23 GMT	4sec
attempt_1430534608975_0001_m_000019_0	SUCCEEDED	map	/default-rack/Slave1:8042	logs	Sat, 02 May 2015 02:45:08 GMT	Sat, 02 May 2015 02:50:39 GMT	5mins, 30sec
attempt_1430534608975_0001_m_000020_1	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:48:48 GMT	Sat, 02 May 2015 02:48:53 GMT	4sec
attempt_1430534608975_0001_m_000021_1	SUCCEEDED	map	/default-rack/Slave3:8042	logs	Sat, 02 May 2015 02:48:03 GMT	Sat, 02 May 2015 02:48:09 GMT	6sec
attempt_1430534608975_0001_m_000022_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:45:38 GMT	Sat, 02 May 2015 02:46:53 GMT	1mins, 14sec
attempt_1430534608975_0001_m_000023_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:46:02 GMT	Sat, 02 May 2015 02:47:08 GMT	1mins, 6sec
attempt_1430534608975_0001_m_000024_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:46:01 GMT	Sat, 02 May 2015 02:47:00 GMT	58sec
attempt_1430534608975_0001_m_000025_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:46:01 GMT	Sat, 02 May 2015 02:46:53 GMT	51sec
attempt_1430534608975_0001_m_000026_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:46:01 GMT	Sat, 02 May 2015 02:47:08 GMT	1mins, 6sec
attempt_1430534608975_0001_m_000027_0	SUCCEEDED	map	/default-rack/Slave2:8042	logs	Sat, 02 May 2015 02:46:55 GMT	Sat, 02 May 2015 02:47:15 GMT	19sec

 

Hadoop還是相當智能的, 她有Speculative Execution機制, 可以預先主動發現一些執行較慢效能較差的節點, 給他們分配的任務就會少一些, 而給效能較好的節點分配較多的任務. 

可以看到, 我們的實驗中, 指派情況為: Slave1-3, Slave2-15, Slave3-10.    Slave1效能較差, 結果分配到的任務最少.

 

實驗2 Combiner的作用

你可能注意到了, 相對於標準的官方WordCount程式, 我們的代碼中加了這麼一行

 job.setCombinerClass(Reduce.class);

 

她是做什麼用的呢?

這 裡要說到MapReduce的Shuffle過程了: 在叢集環境中, Map Task與Reduce Task不在同一節點上, Reduce Task 需要跨節點去其他節點拉取Map Task的結果, 我們可以儘可能的減少Map Task的輸出資料量, 從而達到提高整體效能的目的.

這裡的Combine就是為了合并Map Task的中間結果, 減少Map Task最終的輸出資料量, 減少溢寫到磁碟的資料量, 從而提高效能.

下面我們去掉代碼中這一行，看看任務執行效率如何.

結果跑了20分鐘， 停住不跑了，未能完成....  懷疑是另一物理機抗不住了,導致整個任務卡住了..  現關掉Slave1, 只啟用Slave2和Slave4作DataNode, 繼續測試.

 

開啟Combiner, 結果只花了大概5分鐘

Map Task分布情況:

Slave2 - 16, Slave3 - 12

關閉Combiner,

結果還是跑卡住了... 說明不是跑的慢的節點造成的而是在這種案例中不啟用Combiner甚至是行不通的...後來, 在不啟用Combiner僅在Slave2和Slave3作為DataNode的情況下又進行了多次測試, 幾乎每次都跑卡住了...

這裡至少說明了一點: 在資料量比較大的情況下, 本測試中大概 300M,  不啟用combiner, 會給網路造成極大負載, 甚至導致整個任務無法完成.

 

Combiner到底是什麼，推薦看這篇文章非常詳細的解釋了shuffle過程. http://liouwei20051000285.blog.163.com/blog/static/252367420116208743834/

 

 簡單的說，map階段shuffle過程：

Partition: 決定key/value應該有哪個reducer去處理, 把這種資訊放在緩衝區中

Combiner: 對於相同key的項目做reduce，我們這裡稱之為combiner. 大大減少溢寫到磁碟的資料量和網路傳輸的資料量.

Merge: 將磁碟上的多個溢寫檔案合并為一個溢寫檔案. 這個過程中，如果有相同key也會做combine.

 

 

reduce階段shuffle過程：

 

copy: 拉取資料，把已經完成的map任務複製到reduce任務節點的記憶體區

merge: 與map階段類似，在磁碟產生眾多的溢寫檔案

input: 不斷的merge，形成最終的reducer輸入檔案

MapReduce效能分析實驗