Spark on Yarn always displays accepted when submitting tasks

Spark on Yarn always displays accepted when you submit a task. After an hour, the task fails. However, no error is reported for the logs displayed on the shell terminal at the time of submission, and no logs are generated in the logs folder. Note: spark on Yarn does not need to start the spark cluster. You only need to configure spark on the machine that submits the task, because the task is executed by hadoop, spark is only responsible for task submission.

The task submission command is

Bin/spark-submit -- class org. Apache. Spark. Examples. javawordcount \
-- Master yarn-client \
-- Num-executors 25 \
-- Executor-memory 1g \
-- Executor-cores 1 \
LIB/spark-examples-1.0.1-hadoop2.2.0.jar \
HDFS: // 192.168.1.11: 9000/test

Spark-env.sh configuration is as follows:

Export spark_jar = HDFS: // 192.168.1.11: 9000/test1/spark-assembly-1.0.1-hadoop2.2.0.jar
Export spark_yarn_app_jar = HDFS: // 192.168.1.11: 9000/test1/spark-examples-1.0.1-hadoop2.2.0.jar
Export hadoop_conf_dir =/hadoop/etc/hadoop
Export spark_executor_instances = 1000
Export spark_executor_cores = 1
Export spark_executor_memory = 10000 m
Export spark_driver_memory = 20000 m
Export spark_yarn_app_name = spark

An error occurred while checking logs:

14/09/04 17:10:44 info rmproxy: connecting to ResourceManager at/0.0.0.0: 8032
14/09/04 17:10:45 info client: retrying connect to server: 0.0.0.0/0.0.0.0: 8032. Already tried 0 time (s); retry policy is retryuptomaximumcountwithfixedsleep (maxretries = 10, sleeptime = 1 seconds)
14/09/04 17:10:46 info client: retrying connect to server: 0.0.0.0/0.0.0.0: 8032. Already tried 1 time (s); retry policy is retryuptomaximumcountwithfixedsleep (maxretries = 10, sleeptime = 1 seconds)
14/09/04 17:10:47 info client: retrying connect to server: 0.0.0.0/0.0.0.0: 8032. Already tried 2 time (s); retry policy is retryuptomaximumcountwithfixedsleep (maxretries = 10, sleeptime = 1 seconds)

ResourceManager is obviously not found. We can solve this problem through any of the following methods:

1 add in spark-env.sh

Export spark_yarn_user_env = "classpath = hadoop-2.3.0/etc/hadoop" (change to your hadoop path)

2. Compile the javawordcount program and configure cluster information in the program.

        SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");        sparkConf.set("mapreduce.framework.name", "yarn");        sparkConf.set("mapreduce.jobtracker.address", "192.168.1.10:9001");        sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");        sparkConf.set("yarn.resourcemanager.admin.address", "192.168.1.10:8033");        sparkConf.set("yarn.resourcemanager.address", "192.168.1.10:8032");        sparkConf.set("yarn.resourcemanager.resource-tracker.address", "192.168.1.10:8031");        sparkConf.set("yarn.resourcemanager.scheduler.address", "192.168.1.10:8030");        sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");

Then Package and use your own jar package when submitting the spark task.

Spark on Yarn always displays accepted when submitting tasks