Implement mapreduce multi-file custom output

In common maprduce, there are usually two stages: map and reduce. Without setting, the calculation result is output as multiple files in part-000, in addition, the number of output files is the same as the number of reduce files, and the file content format cannot be arbitrary. This is not conducive to subsequent result processing.

In hadoop, reduce supports multiple outputs, and the output file name is also controllable. It inherits the multipletextoutputformat class and overwrites the generatefilenameforkey method. If you only want to control the output file name, implement your own lognamemultipletextoutputformat class, set jobconf. setoutputformat (lognamemultipletextoutputformat. Class); but this method is only applicable to hadoop of the old version.
Api. If you want to use a new version of the API interface or customize the format of output content, and so on, you need to rewrite some hadoop APIS by yourself.

 
First, you must construct your own multipleoutputformat class to implement the fileoutputformat class (note that it is the fileoutputformat Of The Org. Apache. hadoop. mapreduce. Lib. Output package)

Import Java. io. dataoutputstream; import Java. io. ioexception; import Java. util. hashmap; import Java. util. iterator; import Org. apache. hadoop. conf. configuration; import Org. apache. hadoop. FS. fsdataoutputstream; import Org. apache. hadoop. FS. path; import Org. apache. hadoop. io. writable; import Org. apache. hadoop. io. writablecomparable; import Org. apache. hadoop. io. compress. compressioncodec; import Org. apache. hadoop. io. c Ompress. gzipcodec; import Org. apache. hadoop. mapreduce. outputcommitter; import Org. apache. hadoop. mapreduce. recordwriter; import Org. apache. hadoop. mapreduce. taskattemptcontext; import Org. apache. hadoop. mapreduce. lib. output. fileoutputcommitter; import Org. apache. hadoop. mapreduce. lib. output. fileoutputformat; import Org. apache. hadoop. util. reflectionutils;/*** this abstract class extends the fileoutputformat, Allowing to write the * output data to different output files. there are three basic use cases for * this class. * created on 2012-07-08 * @ author zhoulongliu * @ Param <k> * @ Param <v> */public abstract class multipleoutputformat <k extends writablecomparable <?>, V extends writable> extends fileoutputformat <K, V> {// interface class. You need to implement generatefilenameforkeyvalue in the calling program to obtain the file name private multirecordwriter writer = NULL; Public recordwriter <K, v> getrecordwriter (taskattemptcontext job) throws ioexception, interruptedexception {If (writer = NULL) {writer = new multirecordwriter (job, gettaskoutputpath (job);} return writer ;} /*** get task output path * @ Param conf * @ r Eturn * @ throws ioexception */private path gettaskoutputpath (taskattemptcontext conf) throws ioexception {path workpath = NULL; outputcommitter committer = super. getoutputcommitter (CONF); If (committer instanceof fileoutputcommitter) {workpath = (fileoutputcommitter) committer ). getworkpath ();} else {path outputpath = super. getoutputpath (CONF); If (outputpath = NULL) {Throw new ioexception ("Undefined job output-path");} workpath = outputpath;} return workpath;}/*** determine the output file name (including the extension) through key, value, and Conf) generate the file output file name based * on the given key and the leaf file name. the default behavior is that the * file name does not depend on the key. ** @ Param key the key of the output data * @ Param name the leaf file name * @ Param conf the configure object * @ return generate D file name */protected abstract string generatefilenameforkeyvalue (K key, V value, configuration conf);/*** implement recordwriter class * (internal class) * @ author zhoulongliu **/public class multirecordwriter extends recordwriter <K, V> {/** recordwriter cache */private hashmap <string, recordwriter <K, v> recordwriters = NULL; private taskattemptcontext job = NULL;/** output directory */private path workpath = NULL; Pu BLIC multirecordwriter (taskattemptcontext job, path workpath) {super (); this. job = job; this. workpath = workpath; recordwriters = new hashmap <string, recordwriter <K, V> () ;}@ override public void close (taskattemptcontext context) throws ioexception, interruptedexception {iterator <recordwriter <K, V> values = This. recordwriters. values (). iterator (); While (values. hasnext () {values. next (). close (Context);} This. recordwriters. clear () ;}@ override public void write (K key, V value) throws ioexception, interruptedexception {// get the output file name string basename = generatefilenameforkeyvalue (Key, value, Job. getconfiguration (); // if there is no file name in recordwriters, it is created. Otherwise, write the value directly. Recordwriter <K, V> RW = This. recordwriters. get (basename); If (RW = NULL) {RW = getbaserecordwriter (job, basename); this. recordwriters. put (basename, RW);} RW. write (Key, value);} // $ {mapred. out. dir}/_ temporary/_ $ {taskid}/$ {namewithextension} private recordwriter <K, V> terminate (taskattemptcontext job, string basename) throws ioexception, interruptedexception {configuration conf = job. Getconfiguration (); // check whether the decoder Boolean iscompressed = getcompressoutput (job); string keyvalueseparator = ","; recordwriter <K, V> recordwriter = NULL; If (iscompressed) {class <? Extends compressioncodec> codecclass = getoutputcompressorclass (job, gzipcodec. class); compressioncodec codec = reflectionutils. newinstance (codecclass, conf); Path file = New Path (workpath, basename + codec. getdefaultextension (); fsdataoutputstream fileout = file. getfilesystem (CONF ). create (file, false); // The Custom outputformat recordwriter = new linerecordwriter <K, V> (New dataoutputstream (codec. createoutputstream (fileout), keyvalueseparator);} else {path file = New Path (workpath, basename); fsdataoutputstream fileout = file. getfilesystem (CONF ). create (file, false); // The Custom outputformat recordwriter I used here = new linerecordwriter <K, V> (fileout, keyvalueseparator);} return recordwriter ;}}}

Then you need to customize a linerecordwriter class to implement recordwriter and customize the output format.

Import Java. io. dataoutputstream; import Java. io. ioexception; import Java. io. unsupportedencodingexception; import Org. apache. hadoop. io. nullwritable; import Org. apache. hadoop. io. text; import Org. apache. hadoop. mapreduce. recordwriter; import Org. apache. hadoop. mapreduce. taskattemptcontext;/***** reconstructor recordwriter class * created on 2012-07-08 * @ author zhoulongliu * @ Param <k> * @ Param <v> */public class Li Nerecordwriter <K, V> extends recordwriter <K, V> {Private Static final string utf8 = "UTF-8"; // defines the character encoding format Private Static final byte [] newline; static {try {newline = "\ n ". getbytes (utf8); // defines the line break} catch (unsupportedencodingexception UEE) {Throw new illegalargumentexception ("can't find" + utf8 + "encoding");} protected dataoutputstream out; private Final byte [] keyvalueseparator; // implements the constructor and outputs Stream object and separator public linerecordwriter (dataoutputstream out, string keyvalueseparator) {This. out = out; try {This. keyvalueseparator = keyvalueseparator. getbytes (utf8);} catch (unsupportedencodingexception UEE) {Throw new illegalargumentexception ("can't find" + utf8 + "encoding");} public linerecordwriter (dataoutputstream out) {This (Out, "\ t");} private void writeobject (Object O) throws ioexcep Tion {If (O instanceof text) {text to = (text) O; out. write (. getbytes (), 0,. getlength ();} else {out. write (O. tostring (). getbytes (utf8);}/*** write the mapreduce key and value to the output stream in a custom format */Public synchronized void write (K key, V value) throws ioexception {Boolean nullkey = Key = NULL | key instanceof nullwritable; Boolean nullvalue = value = NULL | value instanceof nullwritable; If (nullk Ey & nullvalue) {return;} If (! Nullkey) {writeobject (key);} If (! (Nullkey | nullvalue) {out. Write (keyvalueseparator);} If (! Nullvalue) {writeobject (value);} Out. Write (newline);} public synchronized void close (taskattemptcontext context) throws ioexception {out. Close ();}}

Next, you need to rewrite the generatefilenameforkeyvalue method in the multipleoutputformat class to customize the name of the output file to be returned. Here, we use the key value to separate the value of the first field with a comma as the output file name, in this way, the first field with the same value will be output to a file and its value will be used as the file name.

 public static class VVLogNameMultipleTextOutputFormat extends MultipleOutputFormat<Text, NullWritable> {                @Override        protected String generateFileNameForKeyValue(Text key, NullWritable value, Configuration conf) {             String sp[] = key.toString().split(",");            String filename = sp[1];            try {                Long.parseLong(sp[1]);            } catch (NumberFormatException e) {                filename = "000000000000";            }            return filename;        }    }

Finally, we set

Configuration conf = getconf ();
Job job = new job (CONF );
Job. setnumreducetasks (12 );
......
Job. setmapperclass (vvetlmapper. Class );
Job. setreducerclass (etlreducer. Class );
Job. setoutputformatclass (vvlognamemultipletextoutputformat. Class); // you can specify a custom multi-file output class.
Fileinputformat. setinputpaths (job, new path (ARGs [0]);
Fileoutputformat. setoutputpath (job, new path (ARGs [1]);
Fileoutputformat. setcompressoutput (job, true); // sets compression for output results
Fileoutputformat. setoutputcompressorclass (job, lzopcodec. Class); // sets lzo compression for output results

OK, so that you have completed writing multi-file output mapreduce that supports the new hadoop API customization.