Four modes of Spark

1.spark mainly has four kinds of operation modes: Loca, standalone, yarn, Mesos.

1) Local mode: On a single machine, typically used for development testing

2) Standalone mode: completely independent spark cluster, not dependent on other clusters, divided into master and work.

The client registers the app with master, the master sends a message to the work, and then starts Driver,executor,driver responsible for sending the task message to executors.

3) Yarn mode: Rely on the Hadoop cluster, yarn Resource scheduling framework, the application submitted to yarn, in applactionmaster (equivalent to stand alone mode master) run driver, scheduling resources on the cluster, Open Excutor to perform the task.

4) Spark on Mesos mode: Similar to yarn mode, runs on the Mesos cluster (Mesos is the open source Distributed resource management framework under Apache, which is known as the kernel of the distributed system. Mesos was originally developed by Amplab of the University of California, Berkeley, and has been widely used in Twitter. ）

2. Starting mode: Sparkshell

The Spark-shell is controlled by different parameters to determine which mode to use. Two parameters are involved:

--master master_url         Spark://Host:port, Mesos://host:port, yarn, or Local. --deploy-mode Deploy_mode   Whether to launch the driver program locally ("Client") or on one of the                            worker Mach Ines inside the cluster ("cluster")                            (default:client).

1) Local mode

./spark-shell--master Local./spark-shell--master local[2]  # Local run, two worker threads, ideal for local CPU core number

2) Standalone mode

./spark-shell--master spark://192.168.1.10:7077

3) Yarn Mode

./spark-shell--master Yarn./spark-shell--master yarn-client# does not support this mode #./spark-shell--master yarn-cluster./ Spark-shell--master yarn--deploy-mode client# does not support this mode #./spark-shell--master yarn--deploy-mode Cluster

Spark Job Deployment mode:

Specifies the job deployment mode by using custom parameters when submitting job tasks by starting Spark-submit form.

eg

Client mode
Spark-submit--master yarn--deploy-mode client--class xxx--executor-memory 1g--executor-cores 2--num-executors 4 xxx. Jar 1000

------------------------

1.client

The driver program runs on the client side.

2.cluster

The driver program runs on a worker.

Note: Spark-shell can only be started in client mode.

　　

Four modes of Spark