Introduction to the "Go" spark architecture and job execution process

Original link http://www.cnblogs.com/shenh062326/p/3658543.html

Introduction to the Spark architecture and job execution process local mode

The simplest way to run spark is through local mode (that is, pseudo-distributed mode).

The Run command is:./bin/run-example Org.apache.spark.examples.SparkPi Local

Standalone-based spark architecture and job execution flow

In standalone mode, the cluster starts with master and worker, where master is responsible for receiving jobs submitted by the client and managing the worker. Provides a web presence cluster and job information.

Noun Explanation:

1. Roles that exist in standalone mode.

Client: Clients process, responsible for submitting jobs to master.

The master node in the Master:standalone mode is responsible for receiving the client-submitted jobs, managing the worker, and ordering the worker to start driver and executor.

The daemon on the slave node in Worker:standalone mode manages the resources of this node, reports the heartbeat to master on a regular basis, receives commands from master, and initiates driver and executor.

Driver: A spark job runtime includes a Driver process, which is also the main process of the job, responsible for parsing the job, building the stage, and dispatching the task to executor. including Dagscheduler,taskscheduler.

Executor: Where a real job is performed, a cluster typically contains multiple Executor, each Executor receives driver's command launch task, and one Executor can execute one or more tasks.

2. Job-related terminology interpretation

Stage: A spark job typically contains one or more stages.

Task: A stage contains one or more tasks that implement parallel functions through multiple tasks.

Dagscheduler: The implementation breaks the spark job into one or more stages, each stage determines the number of tasks based on the number of partition in the RDD, and then generates the corresponding task set to be placed in TaskScheduler.

TaskScheduler: Implement task assignment to executor.

There are two ways to submit a job, namely driver (master of the job, responsible for parsing the job, generating the stage and dispatching task to, including Dagscheduler) running on the worker, driver running on the client. Next, we introduce the operation principle of the two methods respectively.

Driver running on worker

To execute a job through the Org.apache.spark.deploy.Client class, the job Run command is as follows:

./bin/spark-class org.apache.spark.deploy.Client Launch Spark://host:port File:///jar_url Org.apache.spark.examples.SparkPi Spark://host:port

The job execution flow is shown in 1.

Figure 1

Job Execution Process Description:

1. Client submits job to master
2. Master lets a worker start driver, which is schedulerbackend. The worker creates a Driverrunner thread and Driverrunner initiates the schedulerbackend process.
3. In addition, Master will let the rest of the worker start Exeuctor, which is executorbackend. The worker creates a Executorrunner thread, and Executorrunner initiates the executorbackend process.
4. Executorbackend will register with driver's schedulerbackend after startup. The schedulerbackend process contains dagscheduler, which generates execution plans based on the user program and dispatches execution. For each stage of the task, it will be stored in the TaskScheduler, Executorbackend the task in TaskScheduler to executorbackend execution when reporting to Schedulerbackend.
5. The job ends when all stages are completed.

Driver running on the client

To perform the spark job directly, the job Run command is as follows (example):

./bin/run-example Org.apache.spark.examples.SparkPi Spark://host:port

The job execution flow is shown in 2.

Figure 2

Job Execution Process Description:

1. After the client starts, it runs the user program directly and initiates driver related work: Dagscheduler and Blockmanagermaster.
2. The driver of the client is registered with master.
3. Master also causes the worker to start Exeuctor. The worker creates a Executorrunner thread, and Executorrunner initiates the executorbackend process.
4. Executorbackend will register with driver's schedulerbackend after startup. The driver Dagscheduler parses the job and generates the corresponding stage, and each stage contains a task that is assigned to executor execution by TaskScheduler.
5. The job ends when all stages are completed.

Yarn-based spark architecture and job execution flow

Here spark Appmaster equivalent to Schedulerbackend,executor in standalone mode equals standalone Executorbackend,spark The appmaster includes Dagscheduler and Yarnclusterscheduler.

Spark on yarn's execution flow can refer to the Http://www.csdn.net/article/2013-12-04/2817706--YARN Spark on yarn section.

Here is a brief introduction to the main work of Spark applicationmaster. The code references the run () method in Apache Spark 0.9.0 version Applicationmaster.scala.

The steps are as follows:

1. Set environment variables Spark.local.dir and Spark.ui.port. NodeManager the Local_dirs (yarn_local_dirs) variable is passed when the Applicationmaster is started, the variable is set to the Spark.local.dir value. Subsequent temporary files are stored in this directory.
2. Gets the appattemptid that NodeManager passed to Applicationmaster.
3. Create Amrmclient, which is the communication connection between Applicationmaster and ResourceManager.
4. Start the user program, Startuserclass (), using a thread to invoke the main method of the user program by firing. At this point, the Sparkcontext is initialized in the user program, which contains Dagscheduler and TaskScheduler.
5. Register with the ResourceManager.
6. To ResourceManager application containers, it according to the input data and the requested resources, scheduling executor to the corresponding NodeManager, where the scheduling algorithm will consider the input data locality.

Introduction to the "Go" spark architecture and job execution process