Hadoop reading notes (12) MapReduce Custom sorting

Hadoop Reading Notes series article:http://blog.csdn.net/caicongyang/article/category/2166855

1. Description:

The two columns of data given are sorted first in ascending order of the first column, and the second column in ascending order when the first column is the same

Data format:

333231222111

2. Code Sortapp.java

Package Sort;import Java.io.datainput;import java.io.dataoutput;import java.io.ioexception;import Java.net.URI; Import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.filesystem;import Org.apache.hadoop.fs.path;import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.io.writablecomparable;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.mapreduce.lib.output.textoutputformat;import org.apache.hadoop.mapreduce.lib.partition.hashpartitioner;/** * * <p> * Title:SortApp.java * Package sort * &LT  ;/p> * <p> * Description: When the first column is not the same, ascending; When the first column is the same, the second column is ascending * <p> * @author Tom.cai * @created 2014-12-2 10:29:14 * @version V1.0 * */publicClass Sortapp {private static final String Input_path = "Hdfs://192.168.80.100:9000/sort_input";p rivate static final Stri ng Out_path = "Hdfs://192.168.80.100:9000/sort_out";p ublic static void Main (string[] args) throws Exception { Configuration conf = new configuration (); FileSystem FileSystem = filesystem.get (new URI (Input_path), conf), if (Filesystem.exists (new PATH (Out_path))) { Filesystem.delete (New Path (Out_path), true);} Job Job = new Job (Conf,sortapp.class.getsimplename ());//1.1 Specifies the input file path fileinputformat.setinputpaths (job, Input_path); /Specifies which class is used to format the input file Job.setinputformatclass (textinputformat.class);//1.2 Specifies the custom mapper class Job.setmapperclass ( Mymapper.class);//Specify the type of output <k2,v2> job.setmapoutputkeyclass (newk2.class); Job.setmapoutputvalueclass ( Longwritable.class);//1.3 Specifies the partition class Job.setpartitionerclass (Hashpartitioner.class); job.setnumreducetasks (1);//1.4 Todo sort, partition//1.5 todo (optional) merge//2.2 specify custom reduce class Job.setreducerclass (myreducer.class);//Specify Output <k3,v3> Type of Job.setoutputkeyclass (Longwritable.class); JoB.setoutputvalueclass (longwritable.class);//2.3 specifies where to export Fileoutputformat.setoutputpath (job, New Path (Out_path));// Set the format of the output file class Job.setoutputformatclass (Textoutputformat.class);//Submit the code to Jobtracker execution Job.waitforcompletion (true);} Static class Mymapper extends Mapper<longwritable,text, newk2,longwritable>{@Overrideprotected void Map ( Longwritable key, Text value, Context context) throws IOException, interruptedexception {string[] splied = value.tostring ( ). Split ("\ t"); newK2 k2 = new NewK2 (Long.parselong (splied[0]), Long.parselong (splied[1])); final Longwritable v2 = new Longwritable (Long.parselong (splied[1)); Context.write (K2, V2);}} Static class Myreducer extends Reducer<newk2, longwritable, longwritable, longwritable>{@Overrideprotected void Reduce (sort. Sortapp.newk2 key, iterable<longwritable> value, Context context) throws IOException, Interruptedexception { Context.write (New Longwritable (Key.first), New Longwritable (Key.second));}} Static class NewK2 implements WRITABLECOMPARABLE<newk2>{long first; Long Second;public newK2 (long first, long second) {This.first = First;this.second = Second;} Public newK2 () {} @Overridepublic void ReadFields (Datainput input) throws IOException {This.first = Input.readlong (); . Second = Input.readlong ();} @Overridepublic void Write (DataOutput out) throws IOException {Out.writelong (first); Out.writelong (second);} /** * * * When the first column is not the same, ascending; When the first column is the same, the second column is ascending */@Overridepublic int compareTo (newK2 o) {Long temp = this.first-o.first;if (temp!=0 {return (int) temp;} return (int) (this.second-o.second);} @Overridepublic int hashcode () {return This.first.hashCode () +this.second.hashcode ();} @Overridepublic boolean equals (Object obj) {if (!) ( obj instanceof newK2)) {return false;} newK2 K2 = (newK2) obj;return (This.first = K2.first) && (This.second = K2.second);}}}

Welcome everybody to discuss the study together!

Useful Self-collection!

Record and share, let you and I grow together! Welcome to my other blogs, my blog address: Http://blog.csdn.net/caicongyang

Hadoop reading notes (12) MapReduce Custom sorting