Hadoop reading Notes (ix) MapReduce counter

Hadoop Reading Notes series article: http://blog.csdn.net/caicongyang/article/category/2166855

The role of the 1.MapReduce counter statistics map, reduce, and combiner the number of executions, you can easily judge the code execution Flow 2. MapReduce comes with a counter

14/11/26 22:28:51 INFO mapred. JOBCLIENT:COUNTERS:1914/11/26 22:28:51 INFO mapred. Jobclient:file Output Format Counters 14/11/26 22:28:51 INFO mapred. Jobclient:bytes written=2514/11/26 22:28:51 INFO mapred. JOBCLIENT:FILESYSTEMCOUNTERS14/11/26 22:28:51 INFO mapred. JOBCLIENT:FILE_BYTES_READ=34314/11/26 22:28:51 INFO mapred. JOBCLIENT:HDFS_BYTES_READ=4214/11/26 22:28:51 INFO mapred. JOBCLIENT:FILE_BYTES_WRITTEN=12805614/11/26 22:28:51 INFO mapred. JOBCLIENT:HDFS_BYTES_WRITTEN=2514/11/26 22:28:51 INFO mapred. Jobclient:file Input Format Counters 14/11/26 22:28:51 INFO mapred. Jobclient:bytes read=2114/11/26 22:28:51 INFO mapred. Jobclient:map-reduce framework14/11/26 22:28:51 INFO mapred. Jobclient:map output materialized Bytes=4714/11/26 22:28:51 INFO mapred. Jobclient:map input RECORDS=214/11/26 22:28:51 INFO mapred. Jobclient:reduce Shuffle bytes=014/11/26 22:28:51 INFO mapred. jobclient:spilled records=414/11/26 22:28:51 INFO mapred.Jobclient:map output BYTES=3714/11/26 22:28:51 INFO mapred. Jobclient:total committed heap usage (bytes) =36603494414/11/26 22:28:51 INFO mapred. JOBCLIENT:SPLIT_RAW_BYTES=9714/11/26 22:28:51 INFO mapred. Jobclient:combine input RECORDS=014/11/26 22:28:51 INFO mapred. Jobclient:reduce input RECORDS=214/11/26 22:28:51 INFO mapred. Jobclient:reduce input GROUPS=214/11/26 22:28:51 INFO mapred. Jobclient:combine output RECORDS=014/11/26 22:28:51 INFO mapred. Jobclient:reduce output RECORDS=214/11/26 22:28:51 INFO mapred. Jobclient:map Output records=2

3. Custom Counters

Package Counter;import Java.io.ioexception;import Java.net.uri;import org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.filesystem;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.counter;import Org.apache.hadoop.mapreduce.Job; Import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.mapreduce.lib.output.textoutputformat;import  org.apache.hadoop.mapreduce.lib.partition.hashpartitioner;/** * * <p> * Title:WordCount.java * Package counter * </p> * <p> * Description: Custom Counter * <p> * @author tom.cai * @created 2014-11-26 pm 10:47:32 * @version V1.0 * */public class WordCount {private static final String Input_path = "HDFs:192.168.80.100:9000/hello ";p rivate static final String Out_path =" Hdfs://192.168.80.100:9000/out ";p ublic static void Main (string[] args) throws Exception {configuration conf = new Configuration (); FileSystem FileSystem = filesystem.get (new URI (Input_path), conf); Path Outpath = new Path (Out_path), if (Filesystem.exists (Outpath)) {Filesystem.delete (Outpath, True);} Job Job = new Job (conf, WordCount.class.getSimpleName ()); Fileinputformat.setinputpaths (Job, Input_path); Job.setinputformatclass (Textinputformat.class); Job.setmapperclass (Mymapper.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass ( Longwritable.class); Job.setpartitionerclass (Hashpartitioner.class); job.setnumreducetasks (1); Job.setreducerclass (Myreducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass ( Longwritable.class); Fileoutputformat.setoutputpath (Job, New Path (Out_path)); Job.setoutputformatclass (Textoutputformat.class); Job.waitforcompletion (TRUE);} Static class Mymapper extends Mapper<longwritable, text, text, longwritable> {@Overrideprotected void map (longwritable key, text value, context context) throws Ioexc Eption, interruptedexception {<span style= "color: #ff0000;" >/** * counters are used */counter mycounter = Context.getcounter ("MyCounter", "Hello"), if (Value.tostring (). Contains ("Hello") ) {mycounter.increment (1L);} </span>string[] splited = value.tostring (). Split ("\ T"), for (String word:splited) {context.write (new Text (word), New Longwritable (1));}} Static class Myreducer extends Reducer<text, longwritable, Text, longwritable> {@Overrideprotected void reduce ( Text key, iterable<longwritable> value, Context context) throws IOException, interruptedexception {Long Count = 0l;f or (longwritable times:value) {count + = Times.get ();} Context.write (Key, New Longwritable (count));}}}

3. Customize the counter after output

4/11/26 22:45:38 INFO mapred. Jobclient:counters:20
14/11/26 22:45:38 INFO mapred. Jobclient:file Output Format Counters
14/11/26 22:45:38 INFO mapred. Jobclient:bytes written=25
14/11/26 22:45:38 INFO mapred. Jobclient:mycounter
14/11/26 22:45:38 INFO mapred. jobclient:hello=2
14/11/26 22:45:38 INFO mapred. Jobclient:filesystemcounters
14/11/26 22:45:38 INFO mapred. jobclient:file_bytes_read=343
14/11/26 22:45:38 INFO mapred. Jobclient:hdfs_bytes_read=42
14/11/26 22:45:38 INFO mapred. jobclient:file_bytes_written=128036
14/11/26 22:45:38 INFO mapred. Jobclient:hdfs_bytes_written=25
14/11/26 22:45:38 INFO mapred. Jobclient:file Input Format Counters
14/11/26 22:45:38 INFO mapred. Jobclient:bytes read=21
14/11/26 22:45:38 INFO mapred. Jobclient:map-reduce Framework
14/11/26 22:45:38 INFO mapred. Jobclient:map output materialized bytes=47
14/11/26 22:45:38 INFO mapred. Jobclient:map input records=2
14/11/26 22:45:38 INFO mapred. Jobclient:reduce Shuffle bytes=0
14/11/26 22:45:38 INFO mapred. Jobclient:spilled records=4
14/11/26 22:45:38 INFO mapred. Jobclient:map Output bytes=37
14/11/26 22:45:38 INFO mapred. Jobclient:total committed heap usage (bytes) =366034944
14/11/26 22:45:38 INFO mapred. jobclient:split_raw_bytes=97
14/11/26 22:45:38 INFO mapred. Jobclient:combine input Records=0
14/11/26 22:45:38 INFO mapred. Jobclient:reduce input records=2
14/11/26 22:45:38 INFO mapred. Jobclient:reduce input groups=2
14/11/26 22:45:38 INFO mapred. Jobclient:combine Output Records=0
14/11/26 22:45:38 INFO mapred. Jobclient:reduce Output records=2
14/11/26 22:45:38 INFO mapred. Jobclient:map Output records=2

Welcome everybody to discuss the study together!

Useful Self-collection!

Record and share, let you and I grow together! Welcome to my other blogs, my blog address: Http://blog.csdn.net/caicongyang

Hadoop reading Notes (ix) MapReduce counter