A simple application of the MapReduce custom data type

This article takes mobile phone traffic statistics as an example:

The following fields are included in the log

Now you need to count the mobile phone uplink packets, downlink packets, upstream total traffic, downlink total traffic.

Analysis: You can pass data by using the cell phone number as key above 4 field value.

This requires that you define a data type that encapsulates the 4 fields to be counted, passing and shuffle between map and reduce

Note: As a custom type of key, you need to implement the CompareTo method inside the Writablecomparable

As a custom class of value, you only need to implement the method inside the writable

The custom code is as follows:

/*** * MapReduce Module * *@authorNele **/ Public classMobiledatamapreduceextendsConfiguredImplementsTool {//Map Class    /**     *      * @authorNele **/     Public Static classMobiledatamapperextendsMapper<longwritable, text, text, mobiledatawritable> {         PublicText Outputkey =NewText ();  PublicMobiledatawritable Outputvalue =Newmobiledatawritable (); @Override Public voidmap (longwritable key, Text value, context context)throwsIOException, interruptedexception {System.out.println (key+":"+value); String[] Arr= Value.tostring (). Split ("\ T"); Outputkey.set (arr[1]); Outputvalue.set (long.valueof (arr[6]), long.valueof (arr[7]), long.valueof (arr[8]), long.valueof (arr[9]));        Context.write (Outputkey, Outputvalue); }    }    //Reduce class    /***     *      * @authorNele **/     Public Static classMobiledatareducerextendsReducer<text, Mobiledatawritable, Text, mobiledatawritable> {        PrivateText Outputkey =NewText (); PrivateMobiledatawritable Outputvalue =Newmobiledatawritable (); @Override Public voidReduce (Text key, iterable<mobiledatawritable>values, context context)throwsIOException, interruptedexception {LongUppacknum = 0; LongDownpacknum = 0; LongUppayload = 0; LongDownpayload = 0;  for(mobiledatawritable val:values) {uppacknum+=Val.getuppacknum (); Downpacknum+=Val.getdownpacknum (); Uppayload+=val.getuppayload (); Downpayload+=val.getdownpayload ();           } outputkey.set (key);           Outputvalue.set (Uppacknum, Downpacknum, Uppayload, downpayload);        Context.write (Outputkey, Outputvalue); }    }    //Run Method     Public intRun (string[] args)throwsException {Configuration conf=Super. getconf (); //Create JobJob Job = job.getinstance (conf, This. GetClass (). Getsimplename ()); Job.setjarbyclass ( This. GetClass ()); //Set Input PathPath Inpath =NewPath (args[0]);        Fileinputformat.addinputpath (Job, Inpath); //MapJob.setmapperclass (Mobiledatamapper.class); Job.setmapoutputkeyclass (Text.class); Job.setmapoutputvalueclass (mobiledatawritable.class); //ConbileJob.setcombinerclass (Mobiledatareducer.class); //ReduceJob.setreducerclass (Mobiledatareducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (mobiledatawritable.class); //OutputPath Outpath =NewPath (args[1]);        Fileoutputformat.setoutputpath (Job, Outpath); //Submit        returnJob.waitforcompletion (true) ? 0:1; }     Public Static voidMain (string[] args)throwsException {args=Newstring[] {"Hdfs://bigdata5:8020/user/nele/data/input/http_20130313143750.data",         "Hdfs://bigdata5:8020/user/nele/data/output/output6" }; Configuration conf=NewConfiguration (); intStatus = Toolrunner.run (conf,Newmobiledatamapreduce (), args);    System.exit (status); }}

Now you can use the custom type to make phone traffic statistics code as follows:

/*** * MapReduce Module * *@authorNele **/ Public classMobiledatamapreduceextendsConfiguredImplementsTool {//Map Class    /**     *      * @authorNele **/     Public Static classMobiledatamapperextendsMapper<longwritable, text, text, mobiledatawritable> {         PublicText Outputkey =NewText ();  PublicMobiledatawritable Outputvalue =Newmobiledatawritable (); @Override Public voidmap (longwritable key, Text value, context context)throwsIOException, interruptedexception {System.out.println (key+":"+value); String[] Arr= Value.tostring (). Split ("\ T"); Outputkey.set (arr[1]); Outputvalue.set (long.valueof (arr[6]), long.valueof (arr[7]), long.valueof (arr[8]), long.valueof (arr[9]));        Context.write (Outputkey, Outputvalue); }    }    //Reduce class    /***     *      * @authorNele **/     Public Static classMobiledatareducerextendsReducer<text, Mobiledatawritable, Text, mobiledatawritable> {        PrivateText Outputkey =NewText (); PrivateMobiledatawritable Outputvalue =Newmobiledatawritable (); @Override Public voidReduce (Text key, iterable<mobiledatawritable>values, context context)throwsIOException, interruptedexception {LongUppacknum = 0; LongDownpacknum = 0; LongUppayload = 0; LongDownpayload = 0;  for(mobiledatawritable val:values) {uppacknum+=Val.getuppacknum (); Downpacknum+=Val.getdownpacknum (); Uppayload+=val.getuppayload (); Downpayload+=val.getdownpayload ();           } outputkey.set (key);           Outputvalue.set (Uppacknum, Downpacknum, Uppayload, downpayload);        Context.write (Outputkey, Outputvalue); }    }    //Run Method     Public intRun (string[] args)throwsException {Configuration conf=Super. getconf (); //Create JobJob Job = job.getinstance (conf, This. GetClass (). Getsimplename ()); Job.setjarbyclass ( This. GetClass ()); //Set Input PathPath Inpath =NewPath (args[0]);        Fileinputformat.addinputpath (Job, Inpath); //MapJob.setmapperclass (Mobiledatamapper.class); Job.setmapoutputkeyclass (Text.class); Job.setmapoutputvalueclass (mobiledatawritable.class); //ConbileJob.setcombinerclass (Mobiledatareducer.class); //ReduceJob.setreducerclass (Mobiledatareducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (mobiledatawritable.class); //OutputPath Outpath =NewPath (args[1]);        Fileoutputformat.setoutputpath (Job, Outpath); //Submit        returnJob.waitforcompletion (true) ? 0:1; }     Public Static voidMain (string[] args)throwsException {args=Newstring[] {"Hdfs://bigdata5:8020/user/nele/data/input/http_20130313143750.data",         "Hdfs://bigdata5:8020/user/nele/data/output/output6" }; Configuration conf=NewConfiguration (); intStatus = Toolrunner.run (conf,Newmobiledatamapreduce (), args);    System.exit (status); }}

This allows you to count the data in the data log for the various traffic in the phone

A simple application of the MapReduce custom data type