Start the spark application with Spark-submit

Start spark application with Spark-submit

The Bin/spark-submit script is responsible for building the CLASSPATH (Classpath) that contains the spark and its dependencies, which support different cluster managers and spark supported load modes. /bin/spark-submit \--class--master \--deploy-mode \--conf = \ ... # Other options \ [application-arguments] Some common choices are:- Class: The entry point of your application (such as Org.apache.spark.examples.SparkPi)--master: The master URL for the cluster (such as spark://23.195.26.187:7077) Deploy-mode: Deploy your Driver (cluster) on the worker node or locally as an external client. The default is client. --conf: Arbitrary spark configuration attribute, format is key=value. Application-jar: The path that contains the application and the jar package on which it depends. This URL must be globally visible in the cluster, for example, the hdfs://path that exists on all nodes or the file://path application-arguments: Parameters passed to the main method of the main class Spark-submit all available options: # Run Creator locally on 8 cores/bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master local[8] \/path/ To/examples.jar # Run on a Spark Standalone cluster in client deploy mode./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master spark://207.184.161.138:7077 \--executor-memory 20G \ Total-executor-cores \/path/to/examples.jar \ 1000 # Run on a Spark Standalone cluster in ClustEr deploy mode with supervise./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master spark:// 207.184.161.138:7077 \--deploy-mode cluster--supervise--executor-memory 20G \--total-executor-cores Examples.jar \ 1000 # Run on a YARN cluster export hadoop_conf_dir=xxx/bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-cluster \ # can also be ' yarn-client ' for client mode--executor-memory 20G \--num-executors \/path/to/examples.jar \ 1000 # Run a Python creator on a Spark Standalone. cluster Spark-submit \--master spark://207.184.161.138:7077 \ examples/src/main/python/pi.py \ 1000