Hadoop Java Cross-compilation

System: Ubuntu 14.04
HADOOP version:2.6.0 32bits

After you've installed Hadoop and turned on DFS and yarn, you can see six processes with the JPS check:

14779 DataNode15322 NodeManager14657 NameNode15194 ResourceManager17656 Jps14979 SecondaryNameNode

Next we need to run the Wordcout project to verify that the installation is correct.
Wordcount.java:

 Packageorg.apache.hadoop.mapred;ImportJava.io.IOException;ImportJava.util.ArrayList;ImportJava.util.Iterator;ImportJava.util.List;ImportJava.util.StringTokenizer;ImportOrg.apache.hadoop.conf.Configuration;Importorg.apache.hadoop.conf.Configured;ImportOrg.apache.hadoop.fs.Path;Importorg.apache.hadoop.io.IntWritable;Importorg.apache.hadoop.io.LongWritable;ImportOrg.apache.hadoop.io.Text;ImportOrg.apache.hadoop.mapred.FileInputFormat;ImportOrg.apache.hadoop.mapred.FileOutputFormat;ImportOrg.apache.hadoop.mapred.JobClient;Importorg.apache.hadoop.mapred.JobConf;ImportOrg.apache.hadoop.mapred.MapReduceBase;ImportOrg.apache.hadoop.mapred.Mapper;ImportOrg.apache.hadoop.mapred.OutputCollector;ImportOrg.apache.hadoop.mapred.Reducer;ImportOrg.apache.hadoop.mapred.Reporter;ImportOrg.apache.hadoop.util.Tool;ImportOrg.apache.hadoop.util.ToolRunner;/** * This is a example Hadoop map/reduce application. * It reads the text input files, breaks each line into words * and counts them. The output is a locally sorted list of words and the * count of how often they occurred. * To Run:bin/hadoop jar Build/hadoop-examples.jar wordcount * [-M <i>maps</i>] [-R <i>redu Ces</i>] <i>in-dir</i> <i>out-dir</i> * *Public class WordCount extends configured implements Tool {  /** * Counts the words.   * For each line of input, break the line into words and emit them as * (<B>WORD</B>, <b>1</b>). */public static class mapclass extends mapreducebase Implements Mapper<longwritable, text, text, intwritable> {Private Finalstatic intwritable one =NewIntwritable (1);PrivateText Word =NewText ();                     public void Map (longwritable key, Text value, Outputcollector<text, intwritable> output, Reporter Reporter)throwsIOException {String line = value.tostring (); StringTokenizer ITR =NewStringTokenizer (line); while(Itr.hasmoretokens ())        {Word.set (Itr.nexttoken ());      Output.collect (Word, one); }    }  }/** * A Reducer class that just emits the sum of the input values. */public static class Reduce extends mapreducebase Implements Reducer<text, intwritable, text, intwritable> {public void reduce (text key, ITERATOR&LT;INTWRITABLE&G T Values, Outputcollector<text, intwritable> output, Reporter Reporter)throwsioexception {int sum =0; while(Values.hasnext ())      {sum + = Values.next (). get (); } output.collect (Key,NewIntwritable (sum)); }} static int printusage () {System.out.println ("WordCount [-M <maps>] [-R <reduces>] <input> <output>"); Toolrunner.printgenericcommandusage (System.out);return-1; }/** * The main driver for word Count map/reduce.   * Invoke This method to submit the Map/reduce job.   * @throws IOException When there was communication problems with the * job tracker. */public int run (string[] args)throwsException {jobconf conf =NewJobconf (getconf (), WordCount.class); Conf.setjobname ("WordCount");//The keys are Words (strings)Conf.setoutputkeyclass (Text.class);//The values are counts (ints)Conf.setoutputvalueclass (intwritable.class); Conf.setmapperclass (Mapclass.class); Conf.setcombinerclass (Reduce.class); Conf.setreducerclass (Reduce.class); List<string> Other_args =NewArraylist<string> (); for(int i=0; i < args.length; ++i) {Try{if("-M". Equals (Args[i])) {Conf.setnummaptasks (Integer.parseint (Args[++i])); }Else if("-R". Equals (Args[i])) {Conf.setnumreducetasks (Integer.parseint (Args[++i])); }Else{Other_args.add (args[i]); }      }Catch(NumberFormatException except) {System.out.println ("Error:integer expected instead of"+ args[i]);returnPrintusage (); }Catch(ArrayIndexOutOfBoundsException except) {System.out.println ("error:required parameter missing from"+ args[i-1]);returnPrintusage (); }    }//Make sure there is exactly 2 parameters left.    if(Other_args.size ()! =2) {System.out.println ("Error:wrong Number of parameters:"+ other_args.size () +"instead of 2.");returnPrintusage (); } fileinputformat.setinputpaths (conf, Other_args.get (0)); Fileoutputformat.setoutputpath (Conf,NewPath (Other_args.get (1))); Jobclient.runjob (conf);return 0; } public static void Main (string[] args)throwsException {int res = Toolrunner.run (NewConfiguration (),NewWordCount (), args);  System.exit (RES); }}

It is obvious that the Javac command is compiled directly because there is a lot of error in the jar package without Hadoop.
Looked up some information and found that the location of the different jar packages was slightly different because of the Hadoop version.
In the hadoop2.6.0 installation package, look carefully to find the jar packages that are needed in the Hadoop-2.6.0/share/hadoop sub-directories below:

root@fd-ubuntu:/usr/hadoop/hadoop-2.6.0/share/hadoop# lscommon  hdfs  httpfs  kms  mapreduce  tools  yarn

So we can first add a recursive search environment variable to the jar file under this directory in the last line of/etc/profile.

forin$HADOOP_DEV_HOME/share/hadoop -type ddo    HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${X}done

Then write the makefile that generates the Wordcount.jar:

jj = javacWordCount.jar:org    jar -cvf WordCount.jar orgorg: WordCount.java    $(jj) -cp $(HADOOP_CLASSPATH) WordCount.java -d .clear:    rm -rf org WordCount.jar

Follow the above steps to generate the executable jar of the wordcount and put it into HDFs execution.

Hadoop Java Cross-compilation