MapReduce Processing two orders (partition-sort-group)

MapReduce Two-time sequencing principle

In the map phase, the input dataset is split into small chunks splites using InputFormat defined by Job.setinputformatclass, while InputFormat provides a recordreader implementation.
In this example, when Textinputformat is used, the recordreader that he provides will take the byte offset of the text as key, which is the text of the line as value.
This is why the input to the custom map is <LongWritable,Text>, and then call the map method of the custom map, and a map method of <LongWritable,Text> to the input map.
Note that the output should conform to the output <intpair,intwritable> defined in the custom map; Ultimately, a list<intpair,intwritable> is generated; at the end of the map phase, The Job.setpartitionerclass is called first to partition the list, and each partition maps to a reducer. The key comparison function class ordering for the Job.setsortcomparatorclass setting is called within each partition. As you can see, this is in itself a two-time sort. If the key comparison function class is not set by Job.setsortcomparatorclass, the CompareTo method of the implementation of key is used.

In the reduce phase, reducer receives all map outputs mapped to this reducer, The key comparison function class that is set by Job.setsortcomparatorclass is also called to sort all data pairs, and then begins to construct a value iterator corresponding to the key, which is a grouping function that uses the Job.setgroupingcomparatorclass setting. Class. As long as the comparator compares the same two keys, they belong to the same group, their value is placed in a value iterator, and the key of the iterator uses the first key of all keys belonging to the same group. The final step is to enter Reducer's reduce method, and the input to the reduce method is all (key and his value iterator). Also note that the type of the input and output must be consistent with the declaration in the custom reducer.

Core Summary
1.map The final stage of the partition partition. classes that are typically set using Job.setpartitionerclass, if there are no custom classes, are sorted with the Hashcode () method of key
2. within each partition, call Job.setsortcomparatorclass to set the comparison function class for key to sort, if not the CompareTo method that uses the implementation of key.
3. when reduce receives the data transmitted by all maps, Job.setsortcomparatorclass sets the key comparison function class to sort all data pairs, if not the CompareTo method that uses the implementation of key
4. Immediately followed by grouping function classes using the Job.setgroupingcomparatorclass set, the value of the same key is placed in an iterator

Partition---> Sort (two times)---> Grouping
Partition default is Key's Hashcode ()
Sort the default real key's CompareTo ()

-----------------------------------------

Job.setpartitionerclass (Partitioner p); Set the partition. Default partition when Hashcode ()
Job.setsortcomparatorclass (Rawcomparator c); Compare sorts. Shuffle phase map output, before reduce. The default is the CompareTo () method of key
Job.setgroupingcomparatorclass (Rawcomparator c); Group. Reduce phase

-----------------------------------------

Case

Raw data

2 12:12:34 2_hao123
3 09:10:34 3_baidu
1 15:02:41 1_google
3 22:11:34 3_sougou
1 19:23:23 1_baidu
2 13:56:60 2_soso

Sort the data two times according to the first column and the second column, respectively

1. Partition class

Package Test.mr.seconderysort;import org.apache.hadoop.io.text;/* * Partition class */public classes Partitioner Extendsorg.apache.hadoop.mapreduce.partitioner<stringpart, text> {@Overridepublic int getpartition ( Stringpart key, Text value, int numpartitions) {//TODO auto-generated method Stubreturn Math.Abs (Key.hashcode ())% Numpar Titions;}}

2. Custom map output key class, the original data to sort the two columns as the properties of the JavaBean, implement the Writablecomparable interface, implement the CompareTo () sorting method

The CompareTo () method in the Ps:writablecomparatable interface: In this method, if 1 is returned, the current object is preceded by a row, and 1 is returned, followed by 0, equal.

The CompareTo () method in the String class:

/*
  * COMPARETO () The return value is integer, it is the size of the corresponding character (ASCII code order), if the first character and the parameter of the first character, the end comparison, return the difference between them, if the first character and the first character of the parameter is equal, then the second character and the parameter of the second character to do a comparison, and so on, The length of the character is compared until the comparison character or the character being compared is complete.
  *
  * Example:  String s1 = "abc";
  *     String s2 = "ABCD";
  *     String s3 = "ABCDFG";
  *     String s4 = "1BCDFG";
  *     String s5 = "CDFG";
  *     System.out.println (S1.compareto (S2));//-1 (front equal, S1 length small 1)
  *      System.out.println (S1.compareto (S3)); -3 (front equal, S1 length small 3)
  *     System.out.println (S1.compareto (S4));//48 ("A" ASCII is 97, "1 The ASCII code is 49, so return to the
  *     System.out.println (S1.compareto (S5)); -2 ("A" ASCII code is a, "C" ASCII code is 99, so return-2)
  */

Package Test.mr.seconderysort;import Java.io.datainput;import Java.io.dataoutput;import java.io.IOException;import org.apache.hadoop.io.writablecomparable;/* * Custom Key *//* * If you want to sort your own classes, you can write this class to implement the comparable interface * Then write a Comparato method to specify the order in which the objects of this class are sorted. * In this method, if 1 is returned, the current object is in front, 1 is returned, followed by 0, equal to */public class Stringpart implements writablecomparable<stringpart> {/* * Two-column sort */private string first;private string Second;public string GetFirst () {return first;} public void Setfirst (String first) {This.first = first;} Public String Getsecond () {return second;} public void Setsecond (String second) {this.second = second;} Public Stringpart () {super ();//TODO auto-generated constructor stub}public Stringpart (string First, string second) {Supe R (); This.first = First;this.second = Second;} @Overridepublic void Write (DataOutput out) throws IOException {Out.writeutf (first); Out.writeutf (second);} @Overridepublic void ReadFields (Datainput in) throws IOException {This.first = In.readutf (); this.second = In.readutf ();} /* * Sort *//* * CompareTo () The return value is integer, it is the size of the corresponding character (ASCII code order), if the first character and the first character of the parameter are not equal, the end comparison, returns the * difference between them if the first character and the first character of the argument are equal, The second character is compared to the second character of the argument, and so on, until the comparison character or the character being compared has a side * * Full comparison, then the length of the character is compared. * * Example: String S1 = "abc";  * String s2 = "ABCD";  * String s3 = "ABCDFG";  * String s4 = "1BCDFG";  * String S5 = "CDFG"; * SYSTEM.OUT.PRINTLN (S1.compareto (S2));//-1 (front equal, S1 length small 1) * SYSTEM.OUT.PRINTLN (S1.compareto (S3)); -3 (front equal, S1 length small 3) * SYSTEM.OUT.PRINTLN (S1.compareto (S4)); The ASCII code for "A" is 97, and the ASCII code for "1" is 49, so return to * SYSTEM.OUT.PRINTLN (S1.compareto (S5)); -2 ("A" ASCII code is a, "C" ASCII code is 99, so return-2) */@Overridepublic int compareTo (Stringpart o) {if (!this.first.equals ( O.getfirst ())) {return First.compareto (O.getfirst ());//The CompareTo () method of the string} else {if (!this.second.equals (O.getsecond ()) {return Second.compareto (O.getsecond ()))} else {return 0;}}} @Overridepublic int hashcode () {final int prime = 31;int result = 1;result = Prime * result + (first = null)? 0: First.hashcode ()); result = Prime * result + ((second = = null)? 0:second.hashcode ()); return result; @Overridepublic boolean equals (Object obj) {if (this = = obj) return true;if (obj = = null) return False;if (GetClass ()! = obj . GetClass ()) return false; Stringpart other = (Stringpart) obj;if (first = null) {if (Other.first! = null) return false;} else if (!first.equals (othe R.first)) return false;if (second = = null) {if (Other.second! = null) return false;} else if (!second.equals (Other.second)) r Eturn False;return true;}}

3. Grouping classes (grouped according to the first column of the original data)

Package Test.mr.seconderysort;import Org.apache.hadoop.io.writablecomparable;import org.apache.hadoop.io.writablecomparator;/* * Implement grouping */public class Grouping extends Writablecomparator {protected Grouping () {super (Stringpart.class, True);} @Overridepublic int Compare (writablecomparable A, writablecomparable b) {Stringpart A1 = (stringpart) A; Stringpart B1 = (Stringpart) b;//only need to compare the first field of A1,B1 to see if they belong to the same group return A1.getfirst (). CompareTo (B1.getfirst ());}}

4.Map class

Package Test.mr.seconderysort;import Java.io.ioexception;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.mapper;public class Seconderymap extends Mapper< longwritable, text, Stringpart, text> {@Overrideprotected void map (longwritable key, text value,mapper< Longwritable, Text, Stringpart, Text> Context context) throws IOException, interruptedexception {String line = value.tostring ();  string[] str = line.split ("\ t"), if (str.length = = 3) {Stringpart temp = new Stringpart (str[0], str[1]); Context.write (temp, value);}}}

5.Reduce class

Package Test.mr.seconderysort;import Java.io.ioexception;import Org.apache.hadoop.io.nullwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.reducer;public class seconderyredu extendsReducer< Stringpart, text, nullwritable, text> {private static text part = new text ("------------"); @Overrideprotected void redu CE (stringpart key, iterable<text> Values,reducer<stringpart, Text, nullwritable, Text>. Context context) throws IOException, interruptedexception {context.write (Nullwritable.get (), part); for (Text t:values) {Context.write (Nullwritable.get (), t);}}}

6.job class

Package Test.mr.seconderysort;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.nullwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;public class Seconderymain {public static void main (String[] args) throws Exception {configuration conf = new Configuration (); Job Job = new Job (conf); Job.setjarbyclass (Seconderymain.class); Job.setgroupingcomparatorclass (Grouping.class); Job.setpartitionerclass (Partitioner.class); Job.setmapperclass (Seconderymap.class); Job.setMapOutputKeyClass ( Stringpart.class); Job.setmapoutputvalueclass (Text.class); Job.setreducerclass (Seconderyredu.class); Job.setoutputkeyclass (Nullwritable.class); Job.setoutputvalueclass (Text.class); Fileinputformat.addinputpath (Job, New Path (Args[0])); Fileoutputformat.setoutputpath (Job, New Path (Args[1])); Job.waitforcompletion (true);}} 

MapReduce Processing two orders (partition-sort-group)