win7_64 bit MyEclipse2015 yarn-client submit spark to CDH5.10.0 task error and solution

CDH Version: 5.10.0

IDE Environment: Win7 64-bit MyEclipse2015

Spark mode: Yarn

Commit mode: Yarn-client

Before the same IDE environment, to the alone mode Spark submission task, has been very smooth, today, measured spark on yarn mode, the submission can only be yarn-client mode, the other basic unchanged, just changed mode, resulting in the following error:

Java.io.IOException:Cannot Run Program '/etc/hadoop/conf.cloudera.yarn/topology.py ' (in directory D:\workspace2015\ 5_10_0CDH "): CreateProcess error=2,????? Μ???

Tossing for five hours, finally solved, the method is as follows:

Modify the Core-site.xml in the project, locate the configuration net.topology.script.file.name, comment out its value,

Run again to get the correct results.

The code is as follows:

Package com.byit.test;

Import Java.util.Arrays;

Import org.apache.spark.SparkConf;
Import Org.apache.spark.api.java.JavaPairRDD;
Import Org.apache.spark.api.java.JavaRDD;
Import Org.apache.spark.api.java.JavaSparkContext;
Import org.apache.spark.api.java.function.FlatMapFunction;
Import org.apache.spark.api.java.function.Function;
Import Org.apache.spark.api.java.function.Function2;
Import org.apache.spark.api.java.function.PairFunction;

Import Scala. Tuple2;

Import com.byit.getinfo.IGetConf;

/**
*
* @author Geng Detected
* In Yarn client mode, the Core-site.xml in the Src/main/resources directory will be
* Net.topology.script.file.name value is commented out, remember, remember!!!
*/
public class Sparkyarnclienttest implements igetconf{
static String sendstring = "";
public static void Main (string[] args) throws Exception {
TODO auto-generated Method Stub
Sparkyarnclienttest.run ();
}
public static void Run () throws Exception {
TODO auto-generated Method Stub
System.setproperty ("Hadoop_user_name", Sparkuser);
sparkconf conf = new sparkconf ()
. Setappname ("Sparkyarnclienttest")
. Setmaster ("Yarn-client")
. Set ("Spark.yarn.jar", Hdfsurl + "/test/spark/libs/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar");
Javasparkcontext sc = new Javasparkcontext (conf);
String jar = Jarspath + "/sparkyarnclienttest.jar";
Sc.addjar (jar);
String Inpath = Hdfsurl + "/test/spark/input";
String Outpath = Hdfsurl + "/test/spark/output";
Javardd<string> Word = sc.textfile (Inpath). FlatMap (New Toword ());
javapairrdd<string,integer> Wordpair = Word.maptopair (New Mymapper ());
javapairrdd<string,integer> count = Wordpair.reducebykey (new Myreducer ());
javardd<string> result = Count.map (new toString ());
Result.saveastextfile (Outpath);
Sc.close ();
}
@SuppressWarnings ("Serial")
private static class Toword implements Flatmapfunction<string,string> {
Public iterable<string> Call (String words) throws Exception {
TODO auto-generated Method Stub
Return Arrays.aslist (Words.split (sendstring));
}
}
@SuppressWarnings ("Serial")
private static class Mymapper implements Pairfunction<string,string,integer> {

Public tuple2<string, integer> call (String Word) throws Exception {
TODO auto-generated Method Stub
tuple2<string, integer> MyWord = new tuple2<string, integer> (word,1);
return MyWord;
}
}
@SuppressWarnings ("Serial")
private static class Myreducer implements Function2<integer,integer,integer> {
Public integer Call (integer value1, Integer value2) throws Exception {
TODO auto-generated Method Stub
Integer value = value1 + value2;
return value;
}
}
@SuppressWarnings ("Serial")
private static class ToString implements Function<tuple2<string, integer>,string> {

Public String Call (tuple2<string, integer> result) throws Exception {
TODO auto-generated Method Stub

String myresult = result._1 + "\ T" + result._2;
return myresult;
}
}
}

win7_64 bit MyEclipse2015 yarn-client submit spark to CDH5.10.0 task error and solution