Custom Partitions in MapReduce

The data for the reducer task is derived from the Mapper task, which is to say that the mapper task divides the data and runs on different reducer tasks for different data. The process of dividing data by mapper tasks is called partition. The class responsible for implementing the partitioning data is called Partitioner.

The default partition class is Hashpartitioner, which handles the output of the mapper task, the Getpartition () method has three parameters, and key, value refers to the output of the Mapper task, respectively. Numreducetasks refers to the number of reducer tasks set, the default value is 1. Then the remainder of any integer dividing with 1 is certainly 0. That is to say Getpartition (...) The return value of the method is always 0. That is, the output of the mapper task is always given to a reducer task and can only be exported to a single file.

Based on this analysis, if you want to customize the partition and eventually output to multiple files, the data in the Mapper task should be divided into multiple extents. So, we only need to overwrite the Partitioner class of Getpartition (...), according to certain rules let getpartition (...) The return value of the method is 0,1,2,3 .... and two settings in the drive.

Custom Partitions in MapReduce