Mapreduce programming Summary

(1) It is easier to use key-value to map. Each part is handed over to a maptask, and each part is determined by inputformat (usually fileinputformat) (usually 64 m ),

Each maptask calls n map functions. How many map functions are called?

By job. setinputformatclass (?) Medium? It is determined that the default value is textinputformat. Class. textinputformat uses a row as the parsing object and a row corresponds to the call of a map function.

(2) The key-value is complex on the reduce side, and the second parameter is iterable <?> Object, involving <key, list {value1, value2...}>, which corresponds to a call to the reduce function,

That is to say, a reduce function call will process one key and multiple values,

(3) How does the <key, list {value1, value2...}> input come from?

The mapreduce framework comes with sorting of predefined keys (such as text and longwritable,

Aggregate the same keys from different maptasks into <key, list {value1, value2...}> as the input of the reduce function.

(4) The number of maptasks is determined by the number of shards. What is the decision of reducetask?

After each map function is executed, the getpartition function (hashpartitioner class by default) is called to obtain the partition number. The partition number is written to the disk file with the tail of the partition number, so that the reduce end can be pulled,

The most important parameter numreducetasks in the getpartition function will be determined by job. setnumreducetasks. The default value is 1,

Therefore, if this parameter is not set, the getpartition function returns 0, which corresponds to a reducetask.

(5) When the partition is finished, let's talk about grouping. Partitions are determined on the map end. Compared with each map function, the group is placed on the reduce end. Compared with multiple maptasks, the Group belongs to the partition.

What does grouping affect?

(6) When the output key of map is newk2 and compareto is customized, After grouping,

Sort is performed using the compare (byte [] B1, int S1, int L1, byte [] B2, int S2, int l2) of the group class mygroupingcomparator,

Obtain <key, list {value1, value2...}>.

Attached example:

　　

package examples; import java.io.DataInput;import java.io.DataOutput;import java.io.IOException;import java.net.URI;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.RawComparator;import org.apache.hadoop.io.Text;import org.apache.hadoop.io.WritableComparable;import org.apache.hadoop.io.WritableComparator;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.mapreduce.lib.partition.HashPartitioner;public class GroupApp {static final String INPUT_PATH = "hdfs://192.168.2.100:9000/hello";static final String OUTPUT_PATH = "hdfs://192.168.2.100:9000/out";public static void main(String[] args) throws Exception {Configuration conf = new Configuration();final FileSystem fileSystem = FileSystem.get(new URI(INPUT_PATH), conf);final Path outPath = new Path(OUTPUT_PATH);if(fileSystem.exists(outPath)) {fileSystem.delete(outPath, true);}final Job job = new Job(conf, GroupApp.class.getSimpleName());job.setJarByClass(GroupApp.class);FileInputFormat.setInputPaths(job, INPUT_PATH);job.setInputFormatClass(TextInputFormat.class);job.setMapperClass(MyMapper.class);job.setMapOutputKeyClass(NewK2.class);job.setMapOutputValueClass(LongWritable.class);job.setPartitionerClass(MyPartitoner.class);job.setNumReduceTasks(3);job.setGroupingComparatorClass(MyGroupingComparator.class);job.setReducerClass(MyReducer.class);job.setOutputKeyClass(LongWritable.class);job.setOutputValueClass(LongWritable.class);FileOutputFormat.setOutputPath(job, outPath);job.waitForCompletion(true);}static class MyPartitoner extends HashPartitioner<NewK2, LongWritable> {  public int getPartition(NewK2 key, LongWritable value, int numReduceTasks) {  System.out.println("the getPartition() is called...");  if(key.first == 1) {  return 0 % numReduceTasks;  }  else if(key.first == 2) {  return 1 % numReduceTasks;  }  else {  return 2 % numReduceTasks;  }  }}static class NewK2 implements WritableComparable<NewK2> {Long first = 0L;Long second = 0L;public NewK2(){}public NewK2(long first, long second) {this.first = first;this.second = second;}public void write(DataOutput out) throws IOException {out.writeLong(first);out.writeLong(second);}public void readFields(DataInput in) throws IOException {first = in.readLong();second = in.readLong();}public int compareTo(NewK2 o) {System.out.println("the compareTo() is called...");final long minus = this.first - o.first;if(minus != 0) {return (int)minus;}return (int) (this.second - o.second);}}static class MyGroupingComparator implements RawComparator<NewK2> {public int compare(NewK2 o1, NewK2 o2) {//System.out.println("the compare() is called...");return (int) (o1.first - o2.first);}public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {System.out.println("the compare() is called...");return WritableComparator.compareBytes(b1, s1, 8, b2, s2, 8);}}static class MyMapper extends Mapper<LongWritable, Text, NewK2, LongWritable> {protected void map(LongWritable k1, Text v1, Context ctx) throws IOException, InterruptedException {final String[] splited = v1.toString().split("\t");System.out.println("the map() is called...");NewK2 k2 = new NewK2(Integer.parseInt(splited[0]), Integer.parseInt(splited[1]));LongWritable v2 = new LongWritable(Long.parseLong((splited[1])));ctx.write(k2, v2);//System.out.println("the real map output...");//System.out.println("<"+k2.first+","+v2+">");}}static class MyReducer extends Reducer<NewK2, LongWritable, LongWritable, LongWritable> {long v3 = 0;protected void reduce(NewK2 k2, Iterable<LongWritable> v2s, Context ctx) throws IOException, InterruptedException {System.out.println("the reduce() is called...");for(LongWritable secend : v2s) {v3 = secend.get();System.out.println("<"+k2.first+","+k2.second+">, "+v3+"");}System.out.println("--------------------------------------------");System.out.println("the real reduce output...");System.out.println("<"+k2.first+","+v3+">");ctx.write(new LongWritable(k2.first), new LongWritable(v3));System.out.println("--------------------------------------------");}}}

 

　　

 

Mapreduce programming Summary