Spark-submit submitting the job process

1. Job Submission methods and parameters

Let's take a look at the method submitted with spark submission, which is the excerpt from the official section.

# Run application locally on 8cores./bin/spark-Submit--classOrg.apache.spark.examples.SparkPi--master local[8]   /path/to/Examples.jar100# Run on a Spark standalone cluster./bin/spark-Submit--classOrg.apache.spark.examples.SparkPi--master Spark://207.184.161.138:7077 \--executor-Memory 20G--total-executor-cores 100   /path/to/Examples.jar1000# Run on a YARN clusterexport hadoop_conf_dir=XXX./bin/spark-Submit--classOrg.apache.spark.examples.SparkPi--master yarn-cluster \ # can also be ' yarn-client ' forClient Mode--executor-Memory 20G--num-executors 50   /path/to/Examples.jar1000# Run A Python application on a cluster./bin/spark-Submit--master Spark://207.184.161.138:7077 \examples/src/main/python/pi.py1000

Spark-submit submitting the job process