Hadoop development cycle (II): Write Mapper and reducer programs

Writing a simple mapreduce program requires the following three steps:

 

1) Implement Mapper, process input pairs, and output intermediate results;

 

2) Implement CER, calculate intermediate results, and output the final results;

 

3) define the running job in the main method, define a job, and control how the job runs here.

 

This article uses an example (Word Count statistics) to demonstrate basic mapreduce programming.

0. Import the hadoop jar package

Import jar packages under the hadoop directory and lib directory

1. Write mapper class

The er abstract class is a generic type and has four parameter types, respectively specifying the Input key, input value, output key, and output value of the map function. In the above example, the input key is not used (actually represents the location of the row in the text space, and this is not required, so ignore), and the input value is the same text, the output key is a word, and the output value indicates the number of times a word appears.

Hadoop defines its own set of basic types that can be used for network sequence optimization, rather than using the built-in JAVA types. apache. hadoop. as defined in the IO package, the text type used above is equivalent to the string type of Java, and the intwritable type is equivalent to the integer type of Java.

 

package cn.com.yz.mapreduce;import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable> {// --------------------------------------------------------------------private final static IntWritable one = new IntWritable(1); // initial word number is 1private Text word = new Text(); // word// --------------------------------------------------------------------public void map(Object key, Text value, Context context)throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);} // end while} // end map()} // end class WordCountMapper

 

2. Compile the reduce class

The four form parameter types of the reducer abstract class specify the input and output types of the reduce function. In this example, the input key is a word, and the input value is the number of times the word appears. The number of times the word appears is superimposed and the total number of words and words are output.

 

package cn.com.yz.mapreduce;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;public class WordCountReducer extendsReducer<Text, IntWritable, Text, IntWritable> {// --------------------------------------------------------------------private IntWritable result = new IntWritable();// --------------------------------------------------------------------public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();} //end forresult.set(sum);context.write(key, result);} //end reduce()} //end class WordCountReducer

 

3. Compile the main method

 

package cn.com.yz.mapreduce;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount {public static void main(String[] args) throws Exception {Configuration conf = new Configuration();String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) {System.err.println("Usage: wordcount <in> <out>");System.exit(2);}// end if// set job Job job = new Job(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(WordCountMapper.class);job.setCombinerClass(WordCountReducer.class);job.setReducerClass(WordCountReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);// set input and output pathFileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));        //submit job and wait for fininshingSystem.exit(job.waitForCompletion(true) ? 0 : 1);}// end main()} // end class WordCount

The complexity of hadoop lies in that the job configuration has complex attribute parameters, such as file segmentation policies, sorting policies, the size of the map output memory buffer, and the number of worker threads, understanding these parameters can optimize the running of mapreduce programs in the cluster environment.