Spark-submit the task to yarn for execution

Spark-submit -- name sparksubmit_demo -- class com. luogankun. Spark. wordcount -- masterYarn-Client-- Executor-memory 1g -- total-executor-cores 1/home/spark/data/spark. Jar HDFS: // hadoop000: 8020/hello.txt

Note: hadoop_conf_dir needs to be configured for execution on the submitted yarn.

 

When spark is submitted, the resource application is completed at one time. That is to say, the number of executors required for a specific application is calculated at the beginning, if the entire cluster can meet the requirements at this time, it will be submitted; otherwise, it will wait. If a new node is added to the entire cluster, the running program cannot use these new resources. The rebalance mechanism is missing (from Xu Peng's blog ).

 

For more information, see the official documentation: http://spark.apache.org/docs/latest/submitting-applications.html