In Windows Remote submit task to Hadoop cluster (Hadoop 2.6)

Source: Internet
Author: User

I built a Hadoop2.6 cluster with 3 CentOS virtual machines. I would like to use idea to develop a mapreduce program on Windows7 and then commit to execute on a remote Hadoop cluster. After the unremitting Google finally fix

I started using Hadoop's Eclipse plug-in to execute the job and succeeded, and later discovered that MapReduce was executed locally and was not committed to the cluster at all. I added 4 configuration files for Hadoop and I started having problems.

1:org.apache.hadoop.util.shell$exitcodeexception:/bin/bash:line 0: Fg:no Job Control

Online said to modify the source code, the Hadoop2.6 has merged the patch. I forgot how this mistake was solved.

2:stack trace:exitcodeexception exitcode=1:

3:error:could not find or Load main class Org.apache.hadoop.mapreduce.v2.app.MRAppMaster

4: Error: java.lang.RuntimeException :  java.lang.ClassNotFoundException : Class wordcount$map not found

Follow my steps, these problems can be solved, I use the IDE is IDEA1: Copy Hadoop 4 configuration files placed under the SRC directory: core-site.xml,hdfs-site.xml,log4j.properties, Mapred-site.xml,yarn-site.xml 2: Configuring Mapred-site.xml
<Configuration>    < Property>        <name>Mapreduce.framework.name</name>        <value>Yarn</value>    </ Property>    < Property>        <name>Mapred.remote.os</name>        <value>Linux</value>    </ Property>    < Property>        <name>Mapreduce.app-submission.cross-platform</name>        <value>True</value>    </ Property>    < Property>    <name>Mapreduce.application.classpath</name>    <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h        adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value></ Property>        < Property>        <name>Mapreduce.jobhistory.address</name>        <value>master:10020</value>    </ Property>       < Property>                <name>Mapreduce.jobhistory.webapp.address</name>                <value>master:19888</value>        </ Property></Configuration>

Note Mapreduce.application.classpath must be absolute path, do not engage in what $hadoop_home, I here is an error 3: Modify Yarn-site.xml
  1. <Configuration><!--Site specific YARN configuration Properties -  < Property>        <name>Yarn.nodemanager.aux-services</name>        <value>Mapreduce_shuffle</value>    </ Property>    < Property>        <name>Yarn.resourcemanager.address</name>        <value>master:8032</value>    </ Property>< Property>    <name>Yarn.application.classpath</name>    <value>/opt/hadoop-2.6.0/etc/hadoop,/opt/hadoop-2.6.0/share/hadoop/common/*,/opt/hadoop-2.6.0/share/h        adoop/common/lib/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/*,/opt/hadoop-2.6.0/share/hadoop/hdfs/lib/*, /opt/hadoop-2.6.0/share/hadoop/mapreduce/*,/opt/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,/opt/hadoop-2.6 .0/share/hadoop/yarn/*,/opt/hadoop-2.6.0/share/hadoop/yarn/lib/*</value>  </ Property></Configuration>


Note that Yarn.application.classpath must be the absolute path. $hadoop_home 4: Look under My Code
  1. Package Com.gaoxing.hadoop;import Java.io.ioexception;import java.security.privilegedexceptionaction;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.Path;import Org.apache.hadoop.io.intwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.job;import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import org.apache.hadoop.security.UserGroupInformation; Import Org.apache.hadoop.util.genericoptionsparser;public class WordCount {//Inherit mapper interface, set the input type of map to<Object, Text>//Output type is<Text, Intwritable>Public Static class Map extends Mapper<Object, Text,text,intwritable>{//one Indicates that the word appears once private static intwritable one = new intwritable (1);        Word store cuts down the word private text word = new text ();            public void Map (Object key,text value,context Context) throws ioexception,interruptedexception{//line-cut words for input            StringTokenizer st = new StringTokenizer (value.tostring ());            while (St.hasmoretokens ()) {Word.set (St.nexttoken ());//cut words into Word context.write (word, one); }}}//Inherit the Reducer interface, set the input type of reduce<Text, Intwritable>//Output type is<Text, Intwritable>Public Static class Reduce extends Reducer<Text, Intwritable,text,intwritable>{//result Records the frequency of the word private static intwritable result = new intwritable (); public void reduce (Text key,iterable<intwritable>Values,context Context) throws ioexception,interruptedexception{int sum = 0; of the acquired<Key, Value-list>Calculates the sum of the and for (Intwritable val:values) {sums + = Val.get () of value;            }//Set the frequency to result result.set (sum);        Collection of results context.write (key, result); }}/** * @param args */public static void main (string[] args) throws exception{Configuration Co       NF = new Configuration ();       Conf.set ("Mapred.remote.os", "Linux");       Conf.set ("Yarn.resourcemanager.address", "master:8032");        Conf.set ("Mapreduce.framework.name", "yarn");        Conf.set ("Mapred.jar", "D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar");        Conf.set ("Mapreduce.app-submission.cross-platform", "true");        Job Job = job.getinstance (conf);        Job.setjobname ("test");        Configuration jobs each class Job.setjarbyclass (Wordcount.class);        Job.setmapperclass (Map.class);        Job.setcombinerclass (Reduce.class);        Job.setreducerclass (Reduce.class);        Job.setoutputkeyclass (Text.class); Job.setoutPutvalueclass (Intwritable.class);        Fileinputformat.addinputpath (Job, New Path ("hdfs://master:9000/tmp/hbase-env.sh"));        Fileoutputformat.setoutputpath (Job, New Path ("hdfs://master:9000/tmp/out11"));    System.exit (Job.waitforcompletion (true)? 0:1); }}


conf.Set("Mapred.jar","D:\\ideaprojects\\hadooplearn\\out\\artifacts\\hadoo.jar"); This is the most important sentence, or you will report the 4th question above . one of the functions of idea is to pack when compiling:
 got off work.



From for notes (Wiz)



In Windows Remote submit task to Hadoop cluster (Hadoop 2.6)

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.