Operating principle and architecture of the "reprint" Spark series

Reference http://www.cnblogs.com/shishanyuan/p/4721326.html

1. Spark Run architecture 1.1 Terminology Definitions

The concept of Lapplication:spark application is similar to that in Hadoop MapReduce, which refers to a user-written Spark application,
Contains acode for a driver functionand distributed in the clusterExecutor code that runs on multiple nodesThe driver in Ldriver:spark is the main () function that runs the application above and creates Sparkcontext,
The purpose of creating sparkcontext is to prepare the operating environment for the spark application.
In Spark, the Sparkcontext is responsible for and Clustermanager communication, the application of resources, assignment and monitoring of tasks, etc.
When the executor section is complete, driver is responsible for closing the sparkcontext. Typically, a sparkcontext is used to represent a process running on a worker node for Drive;lexecutor:application, which is responsible for running the task and is responsible for having the data in memory or on disk.
Each application has its own separate batch of executor.
In spark on yarn mode, its process name is coarsegrainedexecutorbackend, similar to the yarnchild in Hadoop mapreduce.
A coarsegrainedexecutorbackend process has and has only one executor object, which is responsible for wrapping the task into Taskrunner and extracting an idle thread from the thread pool to run the task.
The number of tasks that each coarsegrainedexecutorbackend can run concurrently depends on the number of CPUs allocated to it; Lcluster Manager: Refers to the external service that obtains resources on the cluster, currently:?  Standalone:spark Native Resource Management, the master responsible for the allocation of resources; Hadoop yarn: The ResourceManager in yarn is responsible for the allocation of resources;
Lworker: Any node in the cluster that can run application code, similar to the NodeManager node in yarn.
In standalone mode, the worker node that is configured through the slave file is the NodeManager node in spark on yarn mode, and the job: a parallel computation consisting of multiple tasks, often by spark Action spawned, a job contains multiple RDD and various operation;l stages (stage) acting on the corresponding RDD: Each job is split into a number of task sets, each set of tasks is called the stage, or taskset, and a job is divided into multiple stages ; L Task: A task that is sent to a executor;

1.2 Spark Run Basic process

Spark runs the basic process see below

1. Build the operating environment for Spark application (start sparkcontext),Sparkcontext to the resource manager (which can be Standalone,mesos, or YARN) Register and apply to run executor resources;

2. The resource manager allocates Executor resources and starts Standaloneexecutorbackend,executor the operation will be sent to the Explorer with the heartbeat;

3. Sparkcontext constructs a dag diagram, decomposes the dag graph into the Stage, and sends the Taskset to Task Scheduler. Executor applies a task to Sparkcontext ,Task Scheduler issues the task to executor run simultaneously Sparkcontext release the application code to Executor.

4. The task runs on executor and releases all resources.

Spark Run architecture Features:

Each application gets a dedicated executor process that resides during application and runs tasks in a multithreaded manner.
This application isolation mechanism has its advantages, either from a scheduling standpoint (each driver dispatches its own task),
Or from a running point of view (a task from a different application runs in a different JVM).
application cannot share data across applications unless the data is written to an external storage system . Lspark is not related to the resource manager, as long as the executor process can be obtained and can maintain communication with each other. The client submitting the Sparkcontext should be close to the worker node (the node running executor), preferably in the same rack,
Because of the large amount of information exchanged between Sparkcontext and executor during the operation of Spark application;
If you want to run in a remote cluster, it is best to use RPC to submit sparkcontext to the cluster, not to run Sparkcontext from the worker. Ltask uses the optimization mechanism of data locality and speculative execution.

1.2.1 Dagscheduler

Dagscheduler A dag that converts a spark job to stage (Directed acyclic graph has a directed acyclic graph) and finds the least expensive scheduling method based on the relationship between the Rdd and the stage. The stage is then presented to TaskScheduler in the form of Taskset.

1.2.2 TaskScheduler

Dagscheduler determines the ideal location for running tasks and passes this information to the underlying TaskScheduler. In addition, Dagscheduler also handles failures due to shuffle data loss, which may require a re-commit to run before the stage (non-shuffle data loss is caused by a task failure handled by TaskScheduler).

TaskScheduler maintains all taskset, and when Executor sends a heartbeat to driver, TaskScheduler assigns the corresponding task according to the remainder of its resource. In addition, the TaskScheduler maintains the running state of all the tasks and retries the failed task. Demonstrates the role of TaskScheduler:

In different operating modes, the Task Scheduler is as follows:

L Spark on standalone mode is TaskScheduler;

L yarn-client Mode is Yarnclientclusterscheduler

L Yarn-cluster Mode is Yarnclusterscheduler

1.3 Rdd Operating principle

So how does the rdd work in the spark architecture? The overall high level, mainly divided into three steps:

1. Create an RDD object

The 2.DAGScheduler module involves operations to calculate the dependency between the RDD. The dependency between the Rdd forms a DAG

3. Each job is divided into multiple stages, one of the main basis for dividing the stage is whether the input of the current calculation factor is deterministic, and if so, it is divided into the same stage, avoiding the message passing overhead between multiple stage.

Look at how the RDD works by looking at the following example of a-Z first letter to find the total number of different names under the same first letter .

Step 1: Create an RDD above example to remove the last collect is an action that does not create an RDD, the first four transformations will create a new rdd. So the first step is to create all the RDD (five internal information).

Step 2: Create an execution plan Spark is piped as much as possible and based on whether the data is being re-organized (stage) , for example, the GroupBy () transformation in this example divides the entire execution plan into two-phase execution. Eventually a DAG (directed acyclic graph , directed acyclic graph) is produced as a logical execution plan.

Step 3: The scheduling task divides the stages into different tasks (Task) , each of which is the combination of data and computation. All tasks in the current phase will be completed before the next phase is performed. Because the first transition in the next phase must be to reorganize the data, you must wait until all the result data in the current stage is calculated before continuing.

Assuming that there are four file blocks under Hdfs://names in this example, Hadooprdd partitions will have four partitions corresponding to the four block data, and Preferedlocations will indicate the best position of the four blocks. Now, you can create four tasks and dispatch them to the appropriate cluster nodes.

2. Spark's running architecture in different clusters

Spark focuses on building a good ecosystem, which not only supports a variety of external file storage systems, but also provides a variety of cluster operation modes. When deployed on a single machine, it can be run either locally (local) or in pseudo-distributed mode, and when deployed in a distributed cluster, you can choose the standalone mode (the Spark's own mode) according to the actual situation of your cluster. Yarn-client mode or Yarn-cluster mode.

Although the various modes of operation of Spark are different in starting mode, running position and scheduling strategy, their purpose is basically consistent, which is to run and manage the task according to the user's configuration and job requirement in the proper location.

2.1 Spark on standalone running process

Standalone mode is the resource scheduling framework implemented by Spark, and its main nodes are client node, master node and worker node . Where driver can run either on the master node or on the local client side.

Driver runs on the master node when submitting a job with the Spark-shell interactive tool to spark;

When you use the Spark-submit tool to submit a job or run a spark task using the "New Sparkconf.setmanager (" spark://master:7077 ") mode on a development platform such as eclips, Idea, The driver is run on the local client side.

2.2 Spark on Yarn run process

Yarn is a unified resource management mechanism on which multiple sets of computational frameworks can be run. In the current world of big data technology, most companies use spark for data calculations, and other computing frameworks, such as MapReduce, Storm, are used for historical reasons or for performance considerations of unilateral business processing. Spark has developed the operating mode of spark on yarn based on this situation, and because of the good elastic resource management mechanism of yarn, not only is it more convenient to deploy application, but also the service that users run in yarn cluster and application resources are completely isolated. A more practical application value is that yarn can manage multiple services running concurrently in the cluster in a queue.

Spark on yarn mode is divided into two modes based on the location of the driver in the cluster: one is the yarn-client mode and the other is the Yarn-cluster (or Yarn-standalone mode).

2.2.1 Yarn Framework Process

Any combination of frame and yarn must follow yarn's development pattern. Before analyzing the implementation details of spark on yarn, it is necessary to analyze some basic principles of the yarn framework first.

The basic operating flow chart for yarn frames is:

Among them,ResourceManager is responsible for allocating the resources of the cluster to each application, while the basic unit of resource allocation and dispatch is Container, which encapsulates the machine resources such as memory,CPU, disk and network, etc. Each task is assigned a Container, which can only be executed in that Container and use the resources encapsulated by that Container.

NodeManager is a compute node that is primarily responsible for initiating the Container required for application , monitoring the use of resources (memory,CPU, disk, network, etc.) and reporting it to ResourceManager.

ResourceManager and nodemanagers together constitute the entire data computing framework,Applicationmaster related to specific application , primarily responsible for negotiating with ResourceManager to obtain appropriate Container, and to track the status of these Container and monitor their progress.

2.2.2 yarn-client

2.2.3 Yarn-cluster

In Yarn-cluster mode, when a user submits an application to yarn, yarn runs the application in two stages: the first stage is to start the driver of Spark as a applicationmaster in the yarn cluster first The second stage is to create the application by Applicationmaster, then request the resource for it to ResourceManager, and start the executor to run the task, while monitoring its entire running process until it finishes running.

The Yarn-cluster workflow is divided into the following steps:

The difference between 2.2.4 Yarn-client and Yarn-cluster

Understand the deep distinction between yarn-client and yarn-cluster before you know a concept:application Master. In yarn, each application instance has a Applicationmaster process , which is the first container that application starts. it is responsible for dealing with ResourceManager and requesting resources , and then telling NodeManager to start container for them. From the deep-seated meaning, the difference between the Yarn-cluster and yarn-client patterns is actually the difference between the applicationmaster process.

L yarn-client mode, driver runs on the client, application master only requests executor,client and the requested container communication to YARN to dispatch their work. That means the client cannot leave.

L yarn-cluster mode, driver runs in AM (Application Master), which is responsible for requesting resources from YARN and overseeing the health of the job. After the user submits the job, the client can be switched off, and the job will continue to run on yarn, so the yarn-cluster mode is not suitable for running interactive types of jobs;

Here are some examples of running in different modes. Slightly.

There is a xxx.cache (), which means that the results of the operation are placed in memory.

Resources:

(1) "Basic concept of Spark1.0.0 operation Architecture" http://blog.csdn.net/book_mmicky/article/details/25714419

(2) "Introduction to the Spark architecture and job execution process" http://www.cnblogs.com/shenh062326/p/3658543.html

(3) "Basic concept of Spark1.0.0 operation Architecture" http://shiyanjun.cn/archives/744.html

Finish

Operating principle and architecture of the "reprint" Spark series