Combiner and Partitioner in Mr

1.combiner

Combiner is a component of the Mr Programming model;

Some tasks in the map may produce a large number of local output, combiner is the role of the map end of the output to do a merge to reduce the data transfer between the map and reduce nodes, improve network IO performance, is one of Mr Optimization means;

Two basic functions:

The aggregation of the key of the output of 1.1map, the key sort of the map output, and the value is iterated;

1.2reduce function.

Does not set the combiner will be executed (in the current cluster is very busy when the settings are not executed);

Combiner execution time: The execution of combiner may be preceded by the merge of the map, which is determined by the configuration option Min.num.spill.for.combine (default 3) . When there are at least 3 spill files generated by the map side, Combiner is executed before the merge operation, otherwise.

In general, you can use the reduce class you write as combiner, but you can also customize it in special cases

1  Public Static classCombinerextendsReducer<text,text,text,text>{2     PrivateText info =NewText ();//prepare to store the new value value in order to split the key value3      Public voidReduce (Text key,iterable<text>values,context Context)throwsIOException, interruptedexception{4       intsum = 0;5        for(Text val:values) {6Sum + =Integer.parseint (val.tostring ());7       }8       intSplitindex = Key.tostring (). IndexOf (":");9Info.set (key.tostring (). substring (splitindex+1) + ":" +sum);//the value of the new valueTenKey.set (key.tostring (). substring (0, Splitindex)); One Context.write (key, info); A     } -}

Job.setcombinerclass (combiner.  Class);

2.partitioner

Combiner can reduce the output of map to the network IO of the Reducer node, but the output of map is assigned to which reducer, which is determined by partitioner;

There is only one way to Partitioner:

int numpartitions)  

The input is the number of map results on <key,value> and reducer, and the number of assigned reducer (integers) is returned. The system default Partitioner is Hashpartitioner, it takes the hash value of key to modulo the number of reducer, obtains the corresponding reducer. This guarantees that the same key value will be assigned to the same reducer.
As with combiner, the default is generally used, but special cases can also be customized

1  Public classNewpartitionextendsHashpartitioner<text,text>{2 String KeyInfo;3        Public intGetpartition (Text key,text value,intnumreducertasks) {4KeyInfo = Key.tostring (). Split (":") [0];5         return Super. Getpartition (NewText (KeyInfo), value, numreducertasks);6       }7}

Job.setpartitionclass (newpartition.  Class);

Combiner and Partitioner in Mr