Partitioner method of the MapReduce partitioning method

Foreword: For two times sort believe everybody also indefinitely, I also is same, to many of these methods do not understand eh, all only temporarily put on one side, when you come into contact with other function, you know the more time you to two order of understanding also is more in depth, at the same time suggest everybody to wordcount the flow to analyze well , to really know what each step is.

What is the role of the 1.Partitioner partitioning class?
2.getPartition () Three parameters what is the difference?
3.numReduceTasks refers to the number of reducer tasks set, what is the default value?
Extended:
If different types of data are assigned to the same partition, will the output data be ordered?

In the case of mapreduce calculations, it is sometimes necessary to divide the final output data into different files, such as in the case of provinces, where data from the same province needs to be placed in a single document, and the same gender data needs to be placed in a file according to gender. We know that the final output data comes from the reducer task. So, if you want to get multiple files, it means that you have the same number of reducer tasks running. The data for the reducer task is derived from the Mapper task, which is to say that the mapper task divides the data and runs on different reducer tasks for different data. The process of dividing data by mapper tasks is called partition. The class responsible for implementing the partitioning data is called Partitioner.

The source code of Partitoner class is as follows:

 PackageOrg.apache.hadoop.mapreduce.lib.partition;ImportOrg.apache.hadoop.mapreduce.Partitioner;/** Partition keys by their {@link object#hashcode ()}. */ Public  class Hashpartitioner<K, V> extends partitioner <K, V> {  /** Use {@link object#hashcode ()} to partition. */   Public intGetpartition (K key, V value,intNumreducetasks) {    //default to use Key's hash value and the maximum value above int to avoid data overflow situation    return(Key.hashcode () & integer.max_value)% Numreducetasks; }}

Hashpartitioner is processing mapper task output, Getpartition () method has three parameters, the source key, value, respectively, refers to the output of the mapper task, Numreducetasks refers to the number of reducer tasks set, the default value is 1. Then the remainder of any integer dividing with 1 is certainly 0. That is to say Getpartition (...) The return value of the method is always 0. That is, the output of the mapper task is always given to a reducer task and can only be exported to a single file. Based on this analysis, if you want to eventually output to multiple files, the data in the Mapper task should be divided into multiple extents. Well, we just have to follow certain rules to make getpartition (...) The return value of the method is 0,1,2,3 ... Can.In Most cases, we will use the default partitioning function, but sometimes we have some, special needs, and need to customize partition to complete our business , the case is as follows:
For the following data, by the length of the string partition, the length of 1 is placed in one, 2 of one, 3 of each.

Henan province; 1 Henan; 2 China; 3 Chinese; 4 large; 1 small; 3 medium; 11

At this time, we use the default partition function, it is not, so we need to customize their own partition, first of all, we need 3 partition output, so in the number of reduce, must be set to 3, followed in partition, when partitioning, To partition according to the length of the specific partition, rather than based on the hash code of the string. The core code is as follows:

    Public Static  class ppartition extends partitioner<text, text>{      @Override     Public intGetpartition (text arg0, text arg1,intARG2) {       /**             * Custom partition, implementation of different lengths of strings, divided into different reduce inside *             * Now   there are only 3 lengths of string, so you can set the number of reduce            to 3 * There are several partitions, set to a few              * */            String key=arg0.tostring ();      if(key.length () = =1){        return 1%ARG2;      }Else if(key.length () = =2){        return 2%ARG2;      }Else if(key.length () = =3){        return 3%ARG2;      }     return  0;    }        }

When running a mapreduce program, simply add the following two lines of code to the main function:

Job.setpartitionerclass (Ppartition.class); Job.setnumreducetasks (3); //Set to 3

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Partitioner method of the MapReduce partitioning method