MapReduce效能分析實驗

來源:互聯網
上載者:User

標籤:

最近應項目需要, 對MapReduce進行了一些實驗測試, 記錄如下.

 

測試環境

3台VM虛擬機器, 都是Ubuntu系統, 1G記憶體, Hadoop 2.6.0

1台 NameNode (Master)

3台 DataNode (Slave)

其中Master和2台Slave (Slave2, Slave3) 位於一配置較強的物理機中, 另1Slave (Slave1) 位於一配置較差的物理機.

 

資料準備

共28個文字檔, 每個檔案大概12M, 共約330M的資料

 

其內容大致是

 

實驗1 節點任務分布情況

我們的測試程式就是基本的單詞計數程式.

package MyPackage;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;        public class WordCount {         public static class Map extends Mapper<Object, Text, Text, IntWritable> {    private final static IntWritable one = new IntWritable(1);    private Text word = new Text();            public void map(Object key, Text value, Context context) throws IOException, InterruptedException {        String line = value.toString();        StringTokenizer tokenizer = new StringTokenizer(line);        while (tokenizer.hasMoreTokens()) {            word.set(tokenizer.nextToken());            context.write(word, one);        }    } }   public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {    public void reduce(Text key, Iterable<IntWritable> values, Context context)       throws IOException, InterruptedException {        int sum = 0;        for (IntWritable val : values) {            sum += val.get();        }        context.write(key, new IntWritable(sum));    } }   public static void main(String[] args) throws Exception {    Configuration conf = new Configuration();                Job job = new Job(conf, "wordcount");            job.setJarByClass(WordCount.class);            job.setOutputKeyClass(Text.class);    job.setOutputValueClass(IntWritable.class);        job.setCombinerClass(Reduce.class);        job.setMapperClass(Map.class);    job.setReducerClass(Reduce.class);                FileInputFormat.addInputPath(job, new Path(args[0]));    FileOutputFormat.setOutputPath(job, new Path(args[1]));                job.waitForCompletion(true); }        }

 

實驗結果

總共花了約11分鐘完成任務

共啟動了28個Map Task

28個Map Task在3個Slave上並非平均分布.

Attempt State Status Node Logs Start Time Finish Time Elapsed Time Note
attempt_1430534608975_0001_m_000000_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:47:03 GMT 3mins, 0sec  
attempt_1430534608975_0001_m_000001_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:08 GMT 3mins, 5sec  
attempt_1430534608975_0001_m_000002_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:08 GMT 3mins, 5sec  
attempt_1430534608975_0001_m_000003_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:07 GMT 3mins, 4sec  
attempt_1430534608975_0001_m_000004_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:08 GMT 3mins, 5sec  
attempt_1430534608975_0001_m_000005_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:12 GMT 3mins, 9sec  
attempt_1430534608975_0001_m_000006_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:08 GMT 3mins, 5sec  
attempt_1430534608975_0001_m_000007_0 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:44:03 GMT Sat, 02 May 2015 02:47:00 GMT 2mins, 57sec  
attempt_1430534608975_0001_m_000008_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:31 GMT 1mins, 28sec  
attempt_1430534608975_0001_m_000009_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:55 GMT 1mins, 52sec  
attempt_1430534608975_0001_m_000010_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:59 GMT 1mins, 57sec  
attempt_1430534608975_0001_m_000011_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:54 GMT 1mins, 52sec  
attempt_1430534608975_0001_m_000012_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:31 GMT 1mins, 28sec  
attempt_1430534608975_0001_m_000013_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:44:02 GMT Sat, 02 May 2015 02:45:55 GMT 1mins, 52sec  
attempt_1430534608975_0001_m_000014_1 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:47:18 GMT Sat, 02 May 2015 02:47:23 GMT 4sec  
attempt_1430534608975_0001_m_000015_0 SUCCEEDED map /default-rack/Slave1:8042 logs Sat, 02 May 2015 02:45:08 GMT Sat, 02 May 2015 02:50:42 GMT 5mins, 33sec  
attempt_1430534608975_0001_m_000016_0 SUCCEEDED map /default-rack/Slave1:8042 logs Sat, 02 May 2015 02:45:08 GMT Sat, 02 May 2015 02:50:57 GMT 5mins, 48sec  
attempt_1430534608975_0001_m_000017_1 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:47:48 GMT Sat, 02 May 2015 02:47:52 GMT 4sec  
attempt_1430534608975_0001_m_000018_1 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:48:18 GMT Sat, 02 May 2015 02:48:23 GMT 4sec  
attempt_1430534608975_0001_m_000019_0 SUCCEEDED map /default-rack/Slave1:8042 logs Sat, 02 May 2015 02:45:08 GMT Sat, 02 May 2015 02:50:39 GMT 5mins, 30sec  
attempt_1430534608975_0001_m_000020_1 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:48:48 GMT Sat, 02 May 2015 02:48:53 GMT 4sec  
attempt_1430534608975_0001_m_000021_1 SUCCEEDED map /default-rack/Slave3:8042 logs Sat, 02 May 2015 02:48:03 GMT Sat, 02 May 2015 02:48:09 GMT 6sec  
attempt_1430534608975_0001_m_000022_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:45:38 GMT Sat, 02 May 2015 02:46:53 GMT 1mins, 14sec  
attempt_1430534608975_0001_m_000023_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:46:02 GMT Sat, 02 May 2015 02:47:08 GMT 1mins, 6sec  
attempt_1430534608975_0001_m_000024_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:46:01 GMT Sat, 02 May 2015 02:47:00 GMT 58sec  
attempt_1430534608975_0001_m_000025_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:46:01 GMT Sat, 02 May 2015 02:46:53 GMT 51sec  
attempt_1430534608975_0001_m_000026_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:46:01 GMT Sat, 02 May 2015 02:47:08 GMT 1mins, 6sec  
attempt_1430534608975_0001_m_000027_0 SUCCEEDED map /default-rack/Slave2:8042 logs Sat, 02 May 2015 02:46:55 GMT Sat, 02 May 2015 02:47:15 GMT 19sec  

 

Hadoop還是相當智能的, 她有Speculative Execution機制, 可以預先主動發現一些執行較慢效能較差的節點, 給他們分配的任務就會少一些, 而給效能較好的節點分配較多的任務. 

可以看到, 我們的實驗中, 指派情況為: Slave1-3, Slave2-15, Slave3-10.    Slave1效能較差, 結果分配到的任務最少.

 

實驗2 Combiner的作用

你可能注意到了, 相對於標準的官方WordCount程式, 我們的代碼中加了這麼一行

 job.setCombinerClass(Reduce.class);

 

她是做什麼用的呢?

這 裡要說到MapReduce的Shuffle過程了: 在叢集環境中, Map Task與Reduce Task不在同一節點上, Reduce Task 需要跨節點去其他節點拉取Map Task的結果, 我們可以儘可能的減少Map Task的輸出資料量, 從而達到提高整體效能的目的.

這裡的Combine就是為了合并Map Task的中間結果, 減少Map Task最終的輸出資料量, 減少溢寫到磁碟的資料量, 從而提高效能.

下面我們去掉代碼中這一行,看看任務執行效率如何.

結果跑了20分鐘, 停住不跑了,未能完成....  懷疑是另一物理機抗不住了,導致整個任務卡住了..  現關掉Slave1, 只啟用Slave2和Slave4作DataNode, 繼續測試.

 

開啟Combiner, 結果只花了大概5分鐘

Map Task分布情況:

Slave2 - 16, Slave3 - 12

關閉Combiner,

結果還是跑卡住了... 說明不是跑的慢的節點造成的而是在這種案例中不啟用Combiner甚至是行不通的...後來, 在不啟用Combiner僅在Slave2和Slave3作為DataNode的情況下又進行了多次測試, 幾乎每次都跑卡住了...

這裡至少說明了一點: 在資料量比較大的情況下, 本測試中大概 300M,  不啟用combiner, 會給網路造成極大負載, 甚至導致整個任務無法完成.

 

Combiner到底是什麼,推薦看這篇文章非常詳細的解釋了shuffle過程. http://liouwei20051000285.blog.163.com/blog/static/252367420116208743834/

 

 簡單的說,map階段shuffle過程:

Partition: 決定key/value應該有哪個reducer去處理, 把這種資訊放在緩衝區中

Combiner: 對於相同key的項目做reduce,我們這裡稱之為combiner. 大大減少溢寫到磁碟的資料量和網路傳輸的資料量.

Merge: 將磁碟上的多個溢寫檔案合并為一個溢寫檔案. 這個過程中,如果有相同key也會做combine.

 

 

reduce階段shuffle過程:

 

copy: 拉取資料,把已經完成的map任務複製到reduce任務節點的記憶體區

merge: 與map階段類似,在磁碟產生眾多的溢寫檔案

input: 不斷的merge,形成最終的reducer輸入檔案

MapReduce效能分析實驗

聯繫我們

該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.