Introduction to Spark on yarn two modes of operation

This article is from: Spark on yarn Two modes of operation introduction
Http://www.aboutyun.com/thread-12294-1-1.html
(Source: About Cloud development)

Questions Guide

1.Spark There are several modes in yarn?
2.Yarn cluster mode, the driver program runs in Yarn, where can the application run results be viewed?
3. What steps does the client submit the request to ResourceManager and upload the jar to HDFs with?
4. What should the parameters passed to the app be specified by?
5. In what mode will the results be output to terminal in the end?

Spark has Yarn-cluster and yarn-client two modes of operation in yarn:

1.Yarn Cluster

Spark driver preferred to start as a applicationmaster in the yarn cluster, Each job that the client submits to ResourceManager assigns a unique applicationmaster on the worker node of the cluster.

The Applicationmaster manages the entire life cycle of the application. Because the driver program runs in yarn, there is no need to start spark master/client in advance, and the results of the application can no longer be displayed by the client (viewable in the history server)

, it is best to save the results in HDFs instead of stdout output, and the client terminal shows the simple health of the job as yarn.

by @Sandy Ryza

by Ming Feng @taobao

See the task initialization in more detail in four steps from the output of terminal:

14/09/28 11:24:52 Info rmproxy:connecting to ResourceManager at HDP01/172.19.1.231:803214/09/28 11:24:52 info client:got Cluster Metric info from Applicationsmanager (ASM), number of NODEMANAGERS:414/09/28 11:24:52 info client:queue info. . Queuename:root.default, queuecurrentcapacity:0.0, Queuemaxcapacity:-1.0, Queueapplicationcount= 0, Queuechildqueuecount = 014/09/28 11:24:52 INFO client:max mem capabililty of a single resource in ThisCluster 819214/09/28 11:24:53 INFO client:uploading file:/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0- Cdh5.1.0.jar to HDFs://Hdp01:8020/user/spark/.sparkstaging/application_1411874193696_0003/spark-examples_2.10-1.0.0-cdh5.1.0.jar 14/09/28 11:24:54 INFO client:uploading file:/usr/lib/spark/assembly/lib/ Spark-assembly-1.0.0-cdh5.1.0-hadoop2.3.0-cdh5.1.0.jar to HDFs://hdp01:8020/user/spark/.sparkstaging/application_1411874193696_0003/ Spark-assembly-1.0.0-cdh5.1.0-hadoop2.3.0-cdh5.1.0.jar14/09/28 11:24:55INFO client:setting up the launch environment14/09/28 11:24:55INFO client:setting up container launch context14/09/28 11:24:55 INFO Client:command forStarting the Spark applicationmaster:list ($JAVA _home/bin/java,-server,-xmx512m,-djava.io.tmpdir= $PWD/tmp,- Dspark.master= "spark://hdp01:7077",-dspark.app.name= "Org.apache.spark.examples.SparkPi",- Dspark.eventlog.enabled= "true",-dspark.eventlog.dir= "/user/spark/applicationhistory",-Dlog4j.configuration= Log4j-spark-container.properties, Org.apache.spark.deploy.yarn.ApplicationMaster,--class, Org.apache.spark.examples.SparkPi,--jar, file:/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0- Cdh5.1.0.jar,--executor-memory, 1024x768,--executor-cores, 1,--num-executors, 2, 1>, <log_dir>/stdout, 2>, & Lt log_dir>/stderr)14/09/28 11:24:55INFO client:submitting application to ASM14/09/28 11:24:55INFO yarnclientimpl:submitted Application application_1411874193696_000314/09/28 11:24:56INFO client:application report from Asm:application identifier:application_1411874193696_0003appid:3Clienttoamtoken:NULLappdiagnostics:appmasterhost:n/AappQueue:root.sparkappMasterRpcPort:-1Appstarttime:1411874695327yarnAppState:ACCEPTEDdistributedFinalState:UNDEFINEDappTrackingUrl:http://hdp01:8088/proxy/application_1411874193696_0003/Appuser:spark

1. Submit the request to ResourceManager by the client and upload the jar to HDFs

This period consists of four steps:

a). Connect to RM

b). Obtain information such as Metric,queue and resource from RM ASM (Applicationsmanager).

c). Upload app jar and spark-assembly jar

d). Set the run environment and container context

2.ResourceManager request resources from NodeManager, create Spark applicationmaster (each sparkcontext has a applicationmanager)

3.NodeManager launch the Spark App Master and register with ResourceManager ASM

4.Spark applicationmaster find jar file from HDFs, start Dagscheduler and yarn Cluster Scheduler

5.ResourceManager to ResourceManager ASM Registration request container Resources (INFO yarnclientimpl:submitted application)

6.ResourceManager notifies NodeManager to assign container, which is available to receive reports from ASM about container. (corresponds to one executor per container)

7.Spark Applicationmaster interacts directly with container (executor) to accomplish this distributed task.

It is important to note that:
a). Localdir in Spark will be replaced by Yarn.nodemanager.local-dirs
b). The number of nodes allowed to fail (spark.yarn.max.worker.failures) is twice times the number of executor, with a minimum of 3.
c). Spark_yarn_user_env The environment variables passed to the SPARK process
d). The parameters passed to the app should be specified by –args

II. yarn-client

(yarnclientclusterscheduler) View the file for the corresponding class

In yarn-client mode, driver runs on the client and obtains resources through Applicationmaster to RM. The local driver is responsible for interacting with all executor container and summarizing the final results. Ending the terminal is equivalent to killing the spark application. In general, you need to configure this if the result of the run is only returned to terminal.

After the client's driver submits the application to yarn, yarn will start Applicationmaster and Excutor, and Applicationmaster and executor will run in container, Container default memory is 1g,applicationmaster allocated memory is driver-memory, Executor allocated memory is executor-memory. At the same time, because driver is on the client side, the running results of the program can be displayed on the client, driver in the form of a process named Sparksubmit.

Configuring Yarn-client mode unification requires Hadoop_conf_dir/yarn_conf_dir and Spark_jar variables

Submit a Task test:

Spark-submit--classOrg.apache.spark.examples.SparkPi--deploy-mode client/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0- cdh5.1.0. Jarterminal Output:14/09/28 11:18:34 INFO Client:command forStarting the Spark applicationmaster:list ($JAVA _home/bin/java,-server,-xmx512m,-djava.io.tmpdir= $PWD/tmp,- Dspark.tachyonstore.foldername= "spark-9287f0f2-2e72-4617-a418-e0198626829b",-dspark.eventlog.enabled= "true",- Dspark.yarn.secondary.jars= "",-dspark.driver.host= "Hdp01",-dspark.driver.appuihistoryaddress= "",- Dspark.app.name= "Spark Pi",-dspark.jars= "file:/usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0- Cdh5.1.0.jar ",-dspark.fileserver.uri=" http://172.19.17.231:53558 ",-dspark.eventlog.dir="/user/spark/ Applicationhistory ",-dspark.master=" Yarn-client ",-dspark.driver.port=" 35938 ",-dspark.httpbroadcast.uri=" http:/ /172.19.17.231:43804 ",-dlog4j.configuration=log4j-spark-container.properties, Org.apache.spark.deploy.yarn.ExecutorLauncher,--class, notused,--jar,NULL,--args ' hdp01:35938 ',--executor-memory, 1024x768,--executor-cores, 1,--num-executors, 2, 1>, <log_dir>/stdout, 2>, <log_dir>/stderr)14/09/28 11:18:34INFO client:submitting application to ASM14/09/28 11:18:34INFO yarnclientschedulerbackend:application report from Asm:appmasterrpcport:-1Appstarttime:1411874314198yarnappstate:accepted ...

Finally output the results to terminal

Introduction to Spark on yarn two modes of operation