Hadoop programming tips (8) --- unit testing (unit test)

Required environment:

Jar packages related to hadoop (download the official website release );

Download the JUnit package (latest );

Download the mockito package;

Download the mrunit package;

Download the powermock-mockito package;

The related package is as follows (related download reference: http://download.csdn.net/detail/fansy1990/7690977 ):

Application scenarios:

In general Mr programming of hadoop, we need to verify our business logic, or use this environment when verifying data streams. This environment does not require a real cloud platform, it only verifies the algorithm or code logic to facilitate code debugging.

Instance:

Mapper:

package fz.mrtest;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class SMSCDRMapper extends Mapper<LongWritable, Text, Text, IntWritable> {   private Text status = new Text();  private final static IntWritable addOne = new IntWritable(1);   /**   * Returns the SMS status code and its count   */  protected void map(LongWritable key, Text value, Context context)      throws java.io.IOException, InterruptedException {     //655209;1;796764372490213;804422938115889;6 is the Sample record format    String[] line = value.toString().split(";");    // If record is of SMS CDR    if (Integer.parseInt(line[1]) == 1) {      status.set(line[4]);      context.write(status, addOne);    }  }}

CER:

package fz.mrtest;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;public class SMSCDRReducer extends  Reducer<Text, IntWritable, Text, IntWritable> {   protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws java.io.IOException, InterruptedException {    int sum = 0;    for (IntWritable value : values) {      sum += value.get();    }    context.write(key, new IntWritable(sum));  }}

Test main program:

package fz.mrtest;import java.io.IOException;import java.util.ArrayList;import java.util.List; import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mrunit.mapreduce.MapDriver;import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;import org.junit.Before;import org.junit.Test; public class SMSCDRMapperReducerTest {   MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;  ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;  MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;   @Before  public void setUp() {    SMSCDRMapper mapper = new SMSCDRMapper();    SMSCDRReducer reducer = new SMSCDRReducer();    mapDriver = MapDriver.newMapDriver(mapper);;    reduceDriver = ReduceDriver.newReduceDriver(reducer);    mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);  }   @Test  public void testMapper() throws IOException {    mapDriver.withInput(new LongWritable(), new Text(        "655209;1;796764372490213;804422938115889;6"));    mapDriver.withOutput(new Text("6"), new IntWritable(1));    mapDriver.runTest();  }   @Test  public void testReducer() throws IOException {    List<IntWritable> values = new ArrayList<IntWritable>();    values.add(new IntWritable(1));    values.add(new IntWritable(1));    reduceDriver.withInput(new Text("6"), values);    reduceDriver.withOutput(new Text("6"), new IntWritable(2));    reduceDriver.runTest();  }  @Test  public void testMR() throws IOException{  mapReduceDriver.withInput(new LongWritable(), new Text(        "655209;1;796764372490213;804422938115889;6"));  mapReduceDriver.withInput(new LongWritable(), new Text(        "6552092;1;796764372490213;804422938115889;6"));  mapReduceDriver.withOutput(new Text("6"), new IntWritable(2));  mapReduceDriver.runTest();  }}

(The Code comes from the mrunit official website, and the final test main program adds a test to the entire project.) There are three test methods for the test main program, test the joint tests of Mapper, reducer, and Mapper and reducer respectively.

Conclusion: The use of hadoop unit testing can easily verify the correctness of programming, without the need to use a real environment to verify the correctness of the Code is possible for efficient development. However, in some special cases, you still need to test the code in the real environment. This requires special consideration. However, in general, this unit test environment is applicable to the compiled mr.

Share, grow, and be happy

Reprinted please indicate blog address: http://blog.csdn.net/fansy1990