Sample Multiplefilewordcount Combinefileinputformat

In the old version of samples, the Multifileinputformat under the old api,mapred is now obsolete.

It is now recommended to use the Combineinputformat below mapreduce for processing.

Application Scenarios:

If the number of files is large and a single file is smaller, if you use Fileinputformat for sharding, a shard is generated based on a file.

Each shard is dropped to a maptask, so that the content of maptask processing is too small, and soon completed, the utilization is not high, because Maptask itself started

The time and resource consumption of processing exceeds the time taken by the information processing itself. Recommend a maptask run for at least one minute or so.

Solution:

Use Combinefileinputformat to redefine the Getsplits method, which reduces data transfer based on the splitsize we specify (typically given as blocksize size)

, package multiple small files into a inputsplit. This reduces the number of maptask generated by the framework.

Example:

For example, there are four files under my Englishwords directory, and using the WordCount example to run, the default is to generate 4 Maptask (Maptask that do not consider failure and generate) a reducetask.

2 Maptask were generated using the Legacy API, and a maptask was generated using the new version of the Multiplefilewordcount sample.

An important method that can be overridden in Combinefileinputformat is:

/**    * Specify the maximum size (in bytes) of each split. Each split was   * approximately equal to the specified size.    */  protected void setmaxsplitsize (long  maxsplitsize) {    this. Maxsplitsize=  maxsplitsize;  }

The example also wrote a data structure Wordoffset, because the original only consider a file (a shard a file) in the information, so key is Offset,value is the value of the current row.

Now there will be multiple files in a shard, so the new data structure Wordoffset will indicate which file is offset, which is clearer.

Sometimes we need to define maptask parameters in our projects. This structure is required to implement the writable interface (which can be serialized).

The most important thing to do with Combinefileinputformat is to implement the reader method, and the most important thing in reader is next ().

The basic idea is actually similar to a single file, just in cases where multiple files need to be processed, an index is required to flag which file is being processed.

In general, there will be the following code in Combinereader:

 Public Static classCombinefilelinerecordreaderextendsRecordreader<wordoffset, text> {    Private LongStartoffset;//offset of the chunk;    Private LongEnd//end of the chunk;    Private LongPos//Current Pos    PrivateFileSystem FS; Privatepath Path; PrivateWordoffset Key; PrivateText value; PrivateFsdatainputstream Filein; PrivateLinereader Reader;  PublicCombinefilelinerecordreader (combinefilesplit split, Taskattemptcontext context, Integer index)throwsIOException { This. Path =Split.getpath (index); FS= This. Path.getfilesystem (Context.getconfiguration ());  This. Startoffset =Split.getoffset (index);  This. end = Startoffset +split.getlength (index); BooleanSkipfirstline =false; //Open the fileFilein =Fs.open (path); if(Startoffset! = 0) {Skipfirstline=true; --Startoffset;      Filein.seek (Startoffset); } Reader=NewLinereader (Filein); if(Skipfirstline) {//Skip first line and re-establish "Startoffset".Startoffset + = Reader.readline (NewText (), 0,                    (int) Math.min (Long) Integer.max_value, End-startoffset)); }       This. pos =Startoffset; }

............

Sample Multiplefilewordcount Combinefileinputformat