MapReduce only uses mapper to write data to multiple hbase tables

Source: Internet
Author: User

Using only mapper without reduce can significantly reduce the time it takes for a mapreduce program to run.

Sometimes the program writes data to multiple hbase tables.

So there is a need for title.

The code given below, not the code that can be run, just shows the necessary items to be set in the driver, the interfaces that the Mapper class needs to implement, the parameters that the map function needs, and how the functions are handled internally.

The implementation of the process is more tortuous, only paste code:

classQos2hbasedriverextendsConfiguredImplementstool{Private StaticLogger Logger =loggerfactory. GetLogger (qos2hbasedriver.class); Private Static Final intDefault_num_reduce = 0; /*** Args[0] Input HDFs file path, args[1] Output table*/@Override Public intRun (string[] args)throwsException {Configuration conf=hbaseconfiguration.create (); Conf.set ("Output", args[1]);//output Table 1Conf.set ("Output2", args[2]);//Output Table 2Job Job=job.getinstance (conf); Job.setjobname ("Iplane_qos2hbase"); Job.setmapperclass (qos2hbasemapper.class); Fileinputformat.setinputpaths (Job, args[0]); Job.setmapoutputkeyclass (immutablebyteswritable.class); Job.setmapoutputvalueclass (Put.class); Job.setoutputformatclass (Multitableoutputformat.class);        Tablemapreduceutil.adddependencyjars (Job);        Tablemapreduceutil.adddependencyjars (Job.getconfiguration ()); Job.setjarbyclass (qos2hbase.class); //set the number of reduce to be adjusted        intNumberreducetasks = 0;        Job.setnumreducetasks (Numberreducetasks); Booleanb = Job.waitforcompletion (true); if(!b) {Logger.error ("Job Error!" "); return-1; }        return0; }}/*** @ClassName: Qos2hbasemapper * @Description: Bring the results into the Mapper class of the HBase library *@authorDangzhang * @date 2014-9-16 pm 1:18:49 **/classQos2hbasemapperextendsMapper<longwritable, Text, immutablebyteswritable, put>{    Private StaticLogger Logger =loggerfactory. GetLogger (qos2hbasemapper.class); @Override Public voidmap (longwritable key, Text line, context context)throwsIOException, interruptedexception {String output= Context.getconfiguration (). Get ("Output"); String Output2= Context.getconfiguration (). Get ("Output2"); //Assembly ROWKEY:IP_IPStringBuffer ROWKEYSB = "AAAA"; Put put=NULL; String Family= "D"; String Qualifier= ""; //direct results into HBase        LongTS =System.currenttimemillis (); Put=NewPut (Bytes.tobytes (rowkeysb.tostring ())); Qualifier= "Del"; Put.add (Bytes.tobytes (family), bytes.tobytes (qualifier), TS, bytes.tobytes (values[6]));//assemble a piece of data            if(!Put.isempty ()) {immutablebyteswritable IB=Newimmutablebyteswritable ();                Ib.set (bytes.tobytes (output)); Context.write (IB, put);//save results to HBase table            }                          //Save History Tablerowkeysb.append (Rowkeyseparator). Append (mydate); Put=NewPut (Bytes.tobytes (rowkeysb.tostring ())); Qualifier= "Del"; Put.add (Bytes.tobytes (family), bytes.tobytes (qualifier), TS, bytes.tobytes (values[6]));//assemble a piece of data            if(!Put.isempty ()) {immutablebyteswritable IB=Newimmutablebyteswritable ();                Ib.set (Bytes.tobytes (OUTPUT2)); Context.write (IB, put);//save results to HBase table            }          }}

MapReduce only uses mapper to write data to multiple hbase tables

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.