Test the MapReduce program on the local file system

In the process of developing a MapReduce program, you can first test the program on the local file system, rather than initially on HDFs, which is easier to debug.

with the as an example of the Maxtemperature program in the authoritative Hadoop guide, the entire project includes the following 3 source files, followed by the mapper program, the reducer program, and the Job startup program:

Maxtemperaturemapper.java, Maxtemperaturereducer.java,maxtemperaturedriver.java

The code for Maxtemperaturemapper.java and Maxtemperaturereducer.java can be found in the installation and use of Mrunit.

Maxtemperaturedriver.java

Import Org.apache.hadoop.conf.configured;import Org.apache.hadoop.fs.path;import org.apache.hadoop.io.IntWritable ; Import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.util.tool;import Org.apache.hadoop.util.toolrunner;public class Maxtemperaturedriver extends configured implements Tool {public int run ( String[] args) throws Exception {if (args.length! = 2) {System.err.printf ("Usage:%s [generic options] <input> <ou Tput>\n ", GetClass (). Getsimplename ()); Toolrunner.printgenericcommandusage (system.err); return-1;} Job Job = new Job (getconf (), "Max temperature"), Job.setjarbyclass (GetClass ()); Fileinputformat.addinputpath (Job, New Path (Args[0])); Fileoutputformat.setoutputpath (Job, New Path (args[1)); Job.setmapperclass (Maxtemperaturemapper.class); Job.setcombinerclass (Maxtemperaturereducer.class); Job.setreducerclass (MaxteMperaturereducer.class); Job.setoutputkeyclass (Text.class); Job.setoutputvalueclass (IntWritable.class); return Job.waitforcompletion (True)? 0:1;} public static void Main (string[] args) throws Exception {int exitCode = Toolrunner.run (New Maxtemperaturedriver (), args); S Ystem.exit (ExitCode);}}

The steps to execute the test program in Eclipse are as follows:

Select File maxtemperaturedriver.java--> right-click Run as-->run configurations-->new Launch configuration--> Select arguments to populate the program arguments with the input and output paths in the local file system, for example

File:///home/hadoop/inputfile:///home/hadoop/output

Click Apply--> to click Run.

After execution, you can determine whether the program is correct based on the contents of the output file.

Test the MapReduce program on the local file system