I built a Hadoop2.6 cluster with 3 CentOS virtual machines. I would like to use idea to develop a mapreduce program on Windows7 and then commit to execute on a remote Hadoop cluster. After the unremitting Google finally fix
I started using Hadoop's Eclipse plug-in to execute the job and succeeded, and later discovered that MapReduce was executed locally and was not committed to the cluster at all. I added 4 configuration files for Hadoop and I started having problems.
1:org.apache.hadoop.util.shell$exitcodeexception:/bin/bash:line 0: Fg:no Job Control
Online said to modify the source code, the Hadoop2.6 has merged the patch. I forgot how this mistake was solved.
2:stack trace:exitcodeexception exitcode=1:
3:error:could not find or Load main class Org.apache.hadoop.mapreduce.v2.app.MRAppMaster
4: Error: java.lang.RuntimeException : java.lang.ClassNotFoundException : Class wordcount$map not found
Follow my steps, these problems can be solved, I use the IDE is IDEA1: Copy Hadoop 4 configuration files placed under the SRC directory: core-site.xml,hdfs-site.xml,log4j.properties, Mapred-site.xml,yarn-site.xml 2: Configuring Mapred-site.xml
<Configuration> < Property> <name>Mapreduce.framework.name</name> <value>Yarn</value> </ Property> < Property> <name>Mapred.remote.os</name> <value>Linux</value> </ Property> < Property> <name>Mapreduce.app-submission.cross-platform</name> <value>True</value> </ Property> < Property> <name>Mapreduce.application.classpath</name> <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value></ Property> < Property> <name>Mapreduce.jobhistory.address</name> <value>master:10020</value> </ Property> < Property> <name>Mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </ Property></Configuration>
Note Mapreduce.application.classpath must be absolute path, do not engage in what $hadoop_home, I here is an error 3: Modify Yarn-site.xml
<Configuration><!--Site specific YARN configuration Properties - < Property> <name>Yarn.nodemanager.aux-services</name> <value>Mapreduce_shuffle</value> </ Property> < Property> <name>Yarn.resourcemanager.address</name> <value>master:8032</value> </ Property>< Property> <name>Yarn.application.classpath</name> <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value> </ Property></Configuration>
Note that Yarn.application.classpath must be the absolute path. $hadoop_home 4: Look under My Code
Package Com.gaoxing.hadoop;import Java.io.ioexception;import java.security.privilegedexceptionaction;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.Path;import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import org.apache.hadoop.security.UserGroupInformation; Import Org.apache.hadoop.util.genericoptionsparser;public class WordCount {//Inherit mapper interface, set the input type of map to<Object, Text>//Output type is<Text, Intwritable>Public Static class Map extends Mapper<Object, Text,text,intwritable>{//one Indicates that the word appears once private static intwritable one = new intwritable (1); Word store cuts down the word private text word = new text (); public void Map (Object key,text value,context Context) throws ioexception,interruptedexception{//line-cut words for input StringTokenizer st = new StringTokenizer (value.tostring ()); while (St.hasmoretokens ()) {Word.set (St.nexttoken ());//cut words into Word context.write (word, one); }}}//Inherit the Reducer interface, set the input type of reduce<Text, Intwritable>//Output type is<Text, Intwritable>Public Static class Reduce extends Reducer<Text, Intwritable,text,intwritable>{//result Records the frequency of the word private static intwritable result = new intwritable (); public void reduce (Text key,iterable<intwritable>Values,context Context) throws ioexception,interruptedexception{int sum = 0; of the acquired<Key, Value-list>Calculates the sum of the and for (Intwritable val:values) {sums + = Val.get () of value; }//Set the frequency to result result.set (sum); Collection of results context.write (key, result); }}/** * @param args */public static void main (string[] args) throws exception{Configuration Co NF = new Configuration (); Conf.set ("Mapred.remote.os", "Linux"); Conf.set ("Yarn.resourcemanager.address", "master:8032"); Conf.set ("Mapreduce.framework.name", "yarn"); Conf.set ("Mapred.jar", "D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar"); Conf.set ("Mapreduce.app-submission.cross-platform", "true"); Job Job = job.getinstance (conf); Job.setjobname ("test"); Configuration jobs each class Job.setjarbyclass (Wordcount.class); Job.setmapperclass (Map.class); Job.setcombinerclass (Reduce.class); Job.setreducerclass (Reduce.class); Job.setoutputkeyclass (Text.class); Job.setoutPutvalueclass (Intwritable.class); Fileinputformat.addinputpath (Job, New Path ("hdfs://master:9000/tmp/hbase-env.sh")); Fileoutputformat.setoutputpath (Job, New Path ("hdfs://master:9000/tmp/out11")); System.exit (Job.waitforcompletion (true)? 0:1); }}
conf.Set("Mapred.jar","D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar"); This is the most important sentence, or you will report the 4th question above . one of the functions of idea is to pack when compiling:
got off work.
From for notes (Wiz)
In Windows Remote submit task to Hadoop cluster (Hadoop 2.6)