Some custom functions in MapReduce

1. counters: allows developers to review the running status and various metrics from a global perspective.

Get counter: Conter myConter = config. getConter ("group name", "counter name ");

Set the initial value for the counter: myConter. setValue (initial value );

Added: myConter. increment ();

2. Combiners (Protocol)

Each map generates a large amount of output. The combiner is used to merge the output at the map end to reduce the data volume of reduce, reducing network transmission.

It can only be merged in a local map and cannot be executed across maps. Therefore, reduce is required.

Combiner is optional, because for some logics, the calculation results before use are inconsistent with those after use.



Job. setCombinerClass (MyReduce. class );



3. Partitioner (Group)

1. The default partitioner of mapreduce is HashPartitioner.

2. Custom

Class KpiPartitioner extends Partitioner <Text, KpiWritable> {

@ Override

Public int getPartition (Text key, LongWritable value, int numPartitions ){

Return (key. toString (). length () = 11 )? ;

}

}

Then add

Job. setPartitionerClass (KpiPartitioner. class );

Job. setNumberReduceTasks (2 );


4. Sorting and grouping

1. in the map and reduce phases, k2 is compared, and v2 is not involved in sorting and comparison. If v2 is to be involved in sorting, k2 and v2 need to be assembled into new classes, can be compared as k2.

2. The group is also based on k2.

Class NewGroup implements RawComparator <NewKey> {

/**

* Compare the size of the specified byte sequence in the byte array

* B1: The first array to be compared

* B2: The second array involved in the comparison

*

* S1: Start position of the first byte array to be compared

* S2: Second

*

* L1: Compare the length

*/

@ Override

Public int compare (byte [] b1, int s1, int l1, byte [] b2, int s2, int l2 ){



Return WritableComparator. compareBytes (b1, s1, 8, b2, s2, 8 );

}



@ Override

Public int compare (NewKey o1, NewKey o2 ){

// TODO Auto-generated method stub

Return 0;

}

}

Then in main

Job. setGroupingComparatorClass (NewGroup. class ).