Hadoop reading notes (10) Understanding combiner from a counter in MapReduce

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/2166855

1.combiner

Q: What is combiner:

A: Combiner occurs in the mapper end, the data to the processing, to the reducer end of the data is reduced, transfer time to end, the operation time is shorter, combiner can not boast mapper execution, (only reduce can accept multiple mapper of the task). Not all algorithms are suitable for the normalization process, for example, averaging

2. Code implementation

Wordcount.java

Package Combine;import Java.io.ioexception;import Java.net.uri;import org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.filesystem;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.Mapper; Import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.mapreduce.lib.output.textoutputformat;import  org.apache.hadoop.mapreduce.lib.partition.hashpartitioner;/** * * <p> * Title:WordCount.java * Package counter * </p> * <p> * Description: * Q: What is combiner: * A: Combiner occurs on the mapper side, the data to the processing, so that the data to the reducer is reduced, transfer time to the end,  The job time is shortened, combiner cannot boast mapper execution, * (only reduce can accept multiple mapper tasks) Not many algorithms are suitable for the normalization process, such as averaging * * <p> * @author Tom.cai * @created 2014-11-26 PM 10:47:32 * @version V1.0 * */public class WordCount {private static final String Input_path = "hdfs://192.168.80.1 00:9000/hello ";p rivate static final String Out_path =" Hdfs://192.168.80.100:9000/out ";p ublic static void Main (string[] args) throws Exception {configuration conf = new Configuration (); FileSystem FileSystem = filesystem.get (new URI (Input_path), conf); Path Outpath = new Path (Out_path), if (Filesystem.exists (Outpath)) {Filesystem.delete (Outpath, True);} Job Job = new Job (conf, WordCount.class.getSimpleName ()),//1.1 setting input file Fileinputformat.setinputpaths (job, Input_path); /1.2 Set Input format job.setinputformatclass (textinputformat.class);//Specify Custom mapper class Job.setmapperclass (Mymapper.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (longwritable.class);//1.3 Set partition Job.setpartitionerclass (Hashpartitioner.class); job.setnumreducetasks (1);//1.4 Sort Group//1.5 job.setcombinerclass (Myreducer.class);// 2.2 Set reduce class Job.setreducerclass (Myreducer.class); Job.setoutputkeyclass (text.class); job.seToutputvalueclass (longwritable.class);//2.3 Specifies the output address fileoutputformat.setoutputpath (Job, New Path (Out_path)); Job.setoutputformatclass (Textoutputformat.class); Job.waitforcompletion (true);} Static class Mymapper extends Mapper<longwritable, text, text, longwritable> {@Overrideprotected void map ( Longwritable key, Text value, Context context) throws IOException, interruptedexception {string[] splited = value.tostring (). Split ("\ T"), for (String word:splited) {context.write (new Text (word), new longwritable (1));}}} Static class Myreducer extends Reducer<text, longwritable, Text, longwritable> {@Overrideprotected void reduce ( Text key, iterable<longwritable> value, Context context) throws IOException, interruptedexception {Long Count = 0l;f or (longwritable times:value) {count + = Times.get ();} Context.write (Key, New Longwritable (count));}}}

</pre><p></p><pre>

3. The counter after joining Combiner: 14/12/01 21:26:41 INFO mapred. Jobclient:counters:19
14/12/01 21:26:41 INFO mapred. Jobclient:file Output Format Counters
14/12/01 21:26:41 INFO mapred. Jobclient:bytes written=20
14/12/01 21:26:41 INFO mapred. Jobclient:filesystemcounters
14/12/01 21:26:41 INFO mapred. jobclient:file_bytes_read=346
14/12/01 21:26:41 INFO mapred. Jobclient:hdfs_bytes_read=40
14/12/01 21:26:41 INFO mapred. jobclient:file_bytes_written=128546
14/12/01 21:26:41 INFO mapred. Jobclient:hdfs_bytes_written=20
14/12/01 21:26:41 INFO mapred. Jobclient:file Input Format Counters
14/12/01 21:26:41 INFO mapred. Jobclient:bytes read=20
14/12/01 21:26:41 INFO mapred. Jobclient:map-reduce Framework
14/12/01 21:26:41 INFO mapred. Jobclient:map output materialized bytes=50
14/12/01 21:26:41 INFO mapred. Jobclient:map input records=2
14/12/01 21:26:41 INFO mapred. Jobclient:reduce Shuffle bytes=0
14/12/01 21:26:41 INFO mapred. Jobclient:spilled records=6
14/12/01 21:26:41 INFO mapred. Jobclient:map Output bytes=52
14/12/01 21:26:41 INFO mapred. Jobclient:total committed heap usage (bytes) =532807680
14/12/01 21:26:41 INFO mapred. jobclient:split_raw_bytes=97
14/12/01 21:26:41 INFO mapred. Jobclient:combine input records=4
14/12/01 21:26:41 INFO mapred. Jobclient:reduce input records=3
14/12/01 21:26:41 INFO mapred. Jobclient:reduce input groups=3
14/12/01 21:26:41 INFO mapred. Jobclient:combine Output records=3
14/12/01 21:26:41 INFO mapred. Jobclient:reduce Output records=3
14/12/01 21:26:41 INFO mapred. Jobclient:map Output records=4

4. The counter before the vesting is not added

14/12/01 21:35:27 INFO mapred. Jobclient:counters:19
14/12/01 21:35:27 INFO mapred. Jobclient:file Output Format Counters
14/12/01 21:35:27 INFO mapred. Jobclient:bytes written=20
14/12/01 21:35:27 INFO mapred. Jobclient:filesystemcounters
14/12/01 21:35:27 INFO mapred. jobclient:file_bytes_read=362
14/12/01 21:35:27 INFO mapred. Jobclient:hdfs_bytes_read=40
14/12/01 21:35:27 INFO mapred. jobclient:file_bytes_written=128090
14/12/01 21:35:27 INFO mapred. Jobclient:hdfs_bytes_written=20
14/12/01 21:35:27 INFO mapred. Jobclient:file Input Format Counters
14/12/01 21:35:27 INFO mapred. Jobclient:bytes read=20
14/12/01 21:35:27 INFO mapred. Jobclient:map-reduce Framework
14/12/01 21:35:27 INFO mapred. Jobclient:map output materialized bytes=66
14/12/01 21:35:27 INFO mapred. Jobclient:map input records=2
14/12/01 21:35:27 INFO mapred. Jobclient:reduce Shuffle bytes=0
14/12/01 21:35:27 INFO mapred. Jobclient:spilled records=8
14/12/01 21:35:27 INFO mapred. Jobclient:map Output bytes=52
14/12/01 21:35:27 INFO mapred. Jobclient:total committed heap usage (bytes) =366034944
14/12/01 21:35:27 INFO mapred. jobclient:split_raw_bytes=97
14/12/01 21:35:27 INFO mapred. Jobclient:combine input Records=0
14/12/01 21:35:27 INFO mapred. Jobclient:reduce input records=4
14/12/01 21:35:27 INFO mapred. Jobclient:reduce input groups=3
14/12/01 21:35:27 INFO mapred. Jobclient:combine Output Records=0
14/12/01 21:35:27 INFO mapred. Jobclient:reduce Output records=3
14/12/01 21:35:27 INFO mapred. Jobclient:map Output records=4

5. Summary

From the previous two counter output can be seen: added to the reduction after reducing the input records from 4 to 3, from the mapper end to Reduce the job less, less transmission time, thereby improving the overall operating time.

Welcome everybody to discuss the study together!

Useful Self-collection!

Record and share, let you and I grow together! Welcome to my other blogs, my blog address: Http://blog.csdn.net/caicongyang

Hadoop reading notes (10) Understanding combiner from a counter in MapReduce