Two-time MapReduce sequencing

Source: Internet
Author: User

By default, the results of the map output will be sorted by default on key, but sometimes it takes two orders to sort the key while still ordering the value. Let's say two times the sort

1, two-time sequencing principle

We divide the two orders into the following stages

map start Stage

In the map phase, using the InputFormat defined by Job.setinputformatclass (), the input datasets are split into small chunks split, and InputFormat provides a recordreader implementation. Here we are using Textinputformat, which provides a recordreader that takes the line number of the text as the key, and the text of the line as value. This is why the input of custom Mapper is <LongWritable,Text>. Then call the map method of the custom mapper, and input the <LongWritable,Text> key value pairs to the Mapper map method.

Map final stage

At the end of the map phase, Job.setpartitionerclass () is called to partition the output of this mapper, with each partition mapped to a reducer. Within each partition, the key comparison function class ordering is called Job.setsortcomparatorclass () setting. As you can see, this is in itself a two-time sort. If the key comparison function class is not set by Job.setsortcomparatorclass (), the CompareTo () method implemented using key

Reduce phase

In the reduce phase, when the reduce () method accepts all map outputs mapped to this reduce, it also calls the key comparison function class set by the Job.setsortcomparatorclass () method to sort all the data. It then begins to construct a value iterator corresponding to the key. Use the Job.setgroupingcomparatorclass () method to set the Grouping function class. As long as the comparator compares the same two keys, they belong to the same group, their value is placed in a value iterator, and the key of the iterator uses the first key of all keys belonging to the same group. The last step is to enter Reducer's reduce () method, where the input of the reduce () method is all key and its value iterator, and also note that the type of input and output must be consistent with the declaration in the custom reducer

Next we have an example that gives you an intuitive understanding of the principle of two sorting

Input file Sort.txt content is

40 20

40 10

40 30

40 5

30 30

30 20

30 10

30 40

50 20

50 50

50 10

50 60

The contents of the output file (from small to large) are as follows

30 10

30 20

30 30

30 40

--------

40 5

40 10

40 20

40 30

--------

50 10

50 20

50 50

50 60

From the output of the results can be seen that the key to achieve a small-to-large ordering, while the same key value also achieved from small to large sort, this is the result of two orders

2, two-time sequencing of the specific process

In this example, compare two times. Sort by the first field first, and then the same as the first field, sorted by the second field. Based on this, we can construct a compound class Intpair, which has two fields, first sorting the first field with a partition, and then sorting the second field using the comparison within the partition. The two-order process is divided into the following steps.

1. Custom key

All custom keys should implement Interface writablecomparable, because it is serializable and comparable. The internal methods of the writablecomparable are as follows

Deserialization, converting from binary in stream to Intpairpublic void ReadFields (Datainput in) throws ioexception//serialization, converting Intpair to binary public void sent using streaming Write (DataOutput out)//  key comparison public int compareTo (Intpair o)//  Default partition class Hashpartitioner, use this method public int Hashcode ()//  default implementation public Boolean equals (Object right)

2. Custom Partitioning

Custom partition function Class Firstpartitioner, is the first comparison of key, complete the sorting of all keys

public static class Firstpartitioner extends partitioner< intpair,intwritable>

Use the Setpartitionerclasss () method in the job to set the Partitioner

Job.setpartitionerclasss (Firstpartitioner.class);

3. Key Comparison class

This is the second comparison of key, where all keys are sorted, that is, the first and second sorts are done at the same time in Intpair. The class is a comparator that can be implemented in two ways.

1) Inherit Writablecomparator

public static class Keycomparator extends Writablecomparator

You must have a constructor and overload the following methods

public int Compare (writablecomparable W1, writablecomparable W2)

2) Implement Interface Rawcomparator.

The above two implementations, in the job, you can use the Setsortcomparatorclass () method to set the key comparison class

Job.setsortcomparatorclass (Keycomparator.class);

Note: If you do not use the custom Sortcomparator class, the key is used by default in the CompareTo () method to sort keys.

4. Define the Grouping class function

In the reduce phase, a value iterator corresponding to the Key is constructed, as long as first is the same group and placed in a value iterator. There are two ways to define this comparer.

1) Inherit Writablecomparator

public static class Groupingcomparator extends Writablecomparator

You must have a constructor and overload the following methods

public int Compare (writablecomparable W1, writablecomparable W2)

2) Implement Interface Rawcomparator.

The above two implementations, in the Job, you can set the grouping class by the Setgroupingcomparatorclass () method

Job.setgroupingcomparatorclass (Groupingcomparator.class);

Also note that if the input and output of reduce are not of the same type, then combiner and Reducer cannot share the Reducer class because the output of combiner is the input of reduce. Unless you redefine a combiner.

3. Code implementation

Hadoop's example package comes with a two-time sequencing algorithm for MapReduce, which improves the two ordering in the example package.

Package Com.buaa;import Java.io.datainput;import Java.io.dataoutput;import java.io.ioexception;import org.apache.hadoop.io.writablecomparable;/** * @ProjectName secondarysort* @PackageName com.buaa* @ClassName intpair* @ Description encapsulates the Key/value in the sample data as a whole as a key, implementing the Writablecomparable interface and overriding its methods * @Author Liu Jishu * @Date 2016-06-07 22:31:53*/ public class Intpair implements writablecomparable<intpair>{private int first;private int second;public Intpair () {}public Intpair (int left, int. right) {Set (left, right);} public void Set (int left, int. right) {first = Left;second = right;} @Overridepublic void ReadFields (Datainput in) throws Ioexception{first = In.readint (); second = In.readint ();} @Overridepublic void Write (DataOutput out) throws Ioexception{out.writeint (first); Out.writeint (second);} @Overridepublic int compareTo (Intpair o) {if (first! = O.first) {return first < O.first? -1:1;} else if (second! = O.second) {return second < O.second? -1:1;} Else{return 0;}} @Overridepublic int Hashcode (){return First * 157 + second;} @Overridepublic Boolean equals (Object right) {if (right = = null) return false;if (this = right) return true;if (right Instan Ceof intpair) {Intpair r = (intpair) Right;return R.first = = First && R.second = second;} Else{return false;}} public int GetFirst () {return first;} public int Getsecond () {return second;}}
Package Com.buaa;import Java.io.ioexception;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.filesystem;import Org.apache.hadoop.fs.Path; Import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.Text ; Import Org.apache.hadoop.io.writablecomparable;import Org.apache.hadoop.io.writablecomparator;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.partitioner;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import org.apache.hadoop.mapreduce.lib.output.textoutputformat;/** * @ProjectName secondarysort* @PackageName com.buaa* @ ClassName secondarysort* @Description todo* @Author Liu Jishu * @Date 2016-06-07 22:40:37*/@SuppressWarnings ("deprecation") Public Class Secondarysort {public static class Map extends Mapper<longwritable, Text, Intpair, intwritable> {public void Map (longwritable key, Text value, Context context) throws IOException, interruptedexception {String line = value.tostring ( ); StringTokenizer tokenizer = new StringTokenizer (line), int left = 0;int right = 0;if (Tokenizer.hasmoretokens ()) {left = in Teger.parseint (Tokenizer.nexttoken ()); if (Tokenizer.hasmoretokens ()) right = Integer.parseint (Tokenizer.nexttoken ( ); Context.write (new Intpair (left, right), new intwritable (right));}} /* Custom partition function class Firstpartitioner, partition */public static class Firstpartitioner extends partitioner< according to first in Intpair Intpair, intwritable>{@Overridepublic int getpartition (Intpair key, intwritable value,int numpartitions) {return Math.Abs (Key.getfirst () * 127)% Numpartitions;}} /* Customize the Groupingcomparator class to implement data groupings within a partition */@SuppressWarnings ("Rawtypes") public static class Groupingcomparator extends Writablecomparator{protected Groupingcomparator () {SupeR (Intpair.class, True);} @Overridepublic int Compare (writablecomparable W1, writablecomparable W2) {Intpair ip1 = (intpair) W1;intpair ip2 = (Intpai r) W2;int L = Ip1.getfirst (); int r = Ip2.getfirst (); return L = = r? 0: (L < R -1:1);}}  public static class reduce extends Reducer<intpair, intwritable, Text, intwritable> {public void reduce (Intpair key, Iterable<intwritable> values, Context context) throws IOException, interruptedexception {for (intwritable Val:val UES) {context.write (New Text (Integer.tostring (Key.getfirst ())), Val);}}} public static void Main (string[] args) throws IOException, Interruptedexception, classnotfoundexception {//          Read Config file configuration conf = new configuration ();//Determine if the path exists, if present, delete path MyPath = new Path (args[1]);          FileSystem HDFs = Mypath.getfilesystem (conf);          if (Hdfs.isdirectory (MyPath)) {Hdfs.delete (MyPath, true); } Job Job = new Job (conf, "Secondarysort");//Set main class Job.setjarbyclass (SecOndarysort.class);//input path fileinputformat.setinputpaths (Job, New Path (args[0));// Output path Fileoutputformat.setoutputpath (Job, New Path (args[1]));//Mapperjob.setmapperclass (Map.class);// Reducerjob.setreducerclass (Reduce.class);//partition Function Job.setpartitionerclass (firstpartitioner.class);// Instead of customizing Sortcomparator, this example uses the CompareTo method in Intpair to sort job.setsortcomparatorclass (); Packet function Job.setgroupingcomparatorclass (groupingcomparator.class);//Map Output Key type Job.setmapoutputkeyclass ( Intpair.class);//Map output value type Job.setmapoutputvalueclass (Intwritable.class);// Reduce output Key type Job.setoutputkeyclass (Text.class);//Reduce Output value type Job.setoutputvalueclass (Intwritable.class);// Input format job.setinputformatclass (textinputformat.class);//output Format job.setoutputformatclass (Textoutputformat.class); System.exit (Job.waitforcompletion (true)? 0:1);}}

If you think reading this blog gives you something to gain, you might as well " top "
If you want to find my new blog more easily, you may wish to " subscribe "
If you are interested in what my blog is talking about, please keep following my follow-up blog, I am " Liu Chao-ljc".

This article is copyrighted by the author and Csdn, welcome reprint, but without the consent of the author must retain this paragraph, and in the article page obvious location to the original link, otherwise reserves the right to pursue legal responsibility.

Two-time MapReduce sequencing

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.