In Windows Remote submit task to Hadoop cluster (Hadoop 2.6)

I built a Hadoop2.6 cluster with 3 CentOS virtual machines. I would like to use idea to develop a mapreduce program on Windows7 and then commit to execute on a remote Hadoop cluster. After the unremitting Google finally fix

I started using Hadoop's Eclipse plug-in to execute the job and succeeded, and later discovered that MapReduce was executed locally and was not committed to the cluster at all. I added 4 configuration files for Hadoop and I started having problems.

1:org.apache.hadoop.util.shell$exitcodeexception:/bin/bash:line 0: Fg:no Job Control

Online said to modify the source code, the Hadoop2.6 has merged the patch. I forgot how this mistake was solved.

2:stack trace:exitcodeexception exitcode=1:

3:error:could not find or Load main class Org.apache.hadoop.mapreduce.v2.app.MRAppMaster

4: Error: java.lang.RuntimeException :  java.lang.ClassNotFoundException : Class wordcount$map not found

Follow my steps, these problems can be solved, I use the IDE is IDEA1: Copy Hadoop 4 configuration files placed under the SRC directory: core-site.xml,hdfs-site.xml,log4j.properties, Mapred-site.xml,yarn-site.xml 2: Configuring Mapred-site.xml

<Configuration>    < Property>        <name>Mapreduce.framework.name</name>        <value>Yarn</value>    </ Property>    < Property>        <name>Mapred.remote.os</name>        <value>Linux</value>    </ Property>    < Property>        <name>Mapreduce.app-submission.cross-platform</name>        <value>True</value>    </ Property>    < Property>    <name>Mapreduce.application.classpath</name>    <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h        adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value></ Property>        < Property>        <name>Mapreduce.jobhistory.address</name>        <value>master:10020</value>    </ Property>       < Property>                <name>Mapreduce.jobhistory.webapp.address</name>                <value>master:19888</value>        </ Property></Configuration>

Note Mapreduce.application.classpath must be absolute path, do not engage in what $hadoop_home, I here is an error 3: Modify Yarn-site.xml

3. <Configuration><!--Site specific YARN configuration Properties -  < Property>        <name>Yarn.nodemanager.aux-services</name>        <value>Mapreduce_shuffle</value>    </ Property>    < Property>        <name>Yarn.resourcemanager.address</name>        <value>master:8032</value>    </ Property>< Property>    <name>Yarn.application.classpath</name>    <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h        adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value>  </ Property></Configuration>

   


Note that Yarn.application.classpath must be the absolute path. $hadoop_home 4: Look under My Code

1. Package Com.gaoxing.hadoop;import Java.io.ioexception;import java.security.privilegedexceptionaction;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.Path;import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import org.apache.hadoop.security.UserGroupInformation; Import Org.apache.hadoop.util.genericoptionsparser;public class WordCount {//Inherit mapper interface, set the input type of map to<Object, Text>//Output type is<Text, Intwritable>Public Static class Map extends Mapper<Object, Text,text,intwritable>{//one Indicates that the word appears once private static intwritable one = new intwritable (1);        Word store cuts down the word private text word = new text ();            public void Map (Object key,text value,context Context) throws ioexception,interruptedexception{//line-cut words for input            StringTokenizer st = new StringTokenizer (value.tostring ());            while (St.hasmoretokens ()) {Word.set (St.nexttoken ());//cut words into Word context.write (word, one); }}}//Inherit the Reducer interface, set the input type of reduce<Text, Intwritable>//Output type is<Text, Intwritable>Public Static class Reduce extends Reducer<Text, Intwritable,text,intwritable>{//result Records the frequency of the word private static intwritable result = new intwritable (); public void reduce (Text key,iterable<intwritable>Values,context Context) throws ioexception,interruptedexception{int sum = 0; of the acquired<Key, Value-list>Calculates the sum of the and for (Intwritable val:values) {sums + = Val.get () of value;            }//Set the frequency to result result.set (sum);        Collection of results context.write (key, result); }}/** * @param args */public static void main (string[] args) throws exception{Configuration Co       NF = new Configuration ();       Conf.set ("Mapred.remote.os", "Linux");       Conf.set ("Yarn.resourcemanager.address", "master:8032");        Conf.set ("Mapreduce.framework.name", "yarn");        Conf.set ("Mapred.jar", "D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar");        Conf.set ("Mapreduce.app-submission.cross-platform", "true");        Job Job = job.getinstance (conf);        Job.setjobname ("test");        Configuration jobs each class Job.setjarbyclass (Wordcount.class);        Job.setmapperclass (Map.class);        Job.setcombinerclass (Reduce.class);        Job.setreducerclass (Reduce.class);        Job.setoutputkeyclass (Text.class); Job.setoutPutvalueclass (Intwritable.class);        Fileinputformat.addinputpath (Job, New Path ("hdfs://master:9000/tmp/hbase-env.sh"));        Fileoutputformat.setoutputpath (Job, New Path ("hdfs://master:9000/tmp/out11"));    System.exit (Job.waitforcompletion (true)? 0:1); }}

   


conf.Set("Mapred.jar","D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar"); This is the most important sentence, or you will report the 4th question above . one of the functions of idea is to pack when compiling:
 got off work.

From for notes (Wiz)

In Windows Remote submit task to Hadoop cluster (Hadoop 2.6)