Spark on Yarn Architecture parsing

。

I. Description of the Hadoop yarn component:

We all know that the fundamental idea of yarn refactoring is to separate the two main functional resource managers and task scheduling monitoring of the original jobtracker into individual components. The new schema uses global management of compute resource allocations for all applications. It consists of three components ResourceManager ,nodemanager and Applicationmaster , as well as a core concept Container.

1.ResourceManager (RM)

is the so-called resource Manager, each cluster one, to achieve global resource management and task scheduling. It can handle the request of client to submit calculation job, start and listen applicationmaster, monitor NodeManager, make resource allocation and dispatch. Each application requires a different type of resource, so a different container is required. The resources here include memory, CPU, disk, network, and so on. (for example, using Spark-submit to execute a program jar package, you need to register with ResourceManager, request the appropriate container, resources), where the ResourceManager provides a scheduling policy plug-in, Responsible for assigning cluster resources to multiple queues and applications. (Can be based on existing capacity scheduling and fair scheduling models)

2. NodeManager (NM)

Node Manager, one for each node, to implement monitoring and reporting of nodes. Handles commands from ResourceManager and also handles commands from Applicationmaster, while monitoring resource availability, reporting errors, and managing resource lifecycles. NodeManager is the agent for each machine framework, a container for executing applications, monitoring the application's resource usage (CPU, memory, hard disk, network) and reporting to the scheduler.

3.Applicationmaster (AM)

Application controller, each job or application one, to achieve the application scheduling and resource coordination. Specifically, it makes data segmentation, applies resources for applications, assigns tasks, and accomplishes task monitoring and fault tolerance. In fact, the applicationmaster of each application is a detailed framework library. It combines the resources obtained from ResourceManager and NodeManager to run and listen to tasks. Applicationmaster is responsible for asking ResourceManager for the appropriate resource containers (Containter) to run tasks, track the status of applications and monitor their processes, and handle the reasons for the failure of the task.

4.Container

containers, which encapsulate their resources, including memory, CPU, disk, network, and so on. Each task is assigned a container that can execute only in that container and use the resources encapsulated by that container. When an application makes a resource request, ResourceManager does not immediately return to the resource that satisfies the requirement, and needs to applicationmaster and ResourceManager to communicate continuously to detect that the allocated resources are sufficient to be allocated. Once assigned, Applicationmaster can get the resources represented by container from ResourceManager. (container can be seen as a serializable Java object that contains field information) in general, each container can be used to perform a task. Applicationmaster after receiving one or more container, the container is further assigned to a task inside, and after the task is determined, applicationmaster the task to run the environment (including running commands, environment variables, dependent external files, etc.), together with the resource information in container, is encapsulated into the Containerlaunchcontext object, which communicates with the corresponding NodeManager to initiate the task.

Second, Spark on Yarn

1. When a Spark-submit task is submitted, spark launches the application that the user submitted, starting with driver, in the Startuserclass function that specifically launches a thread (the thread named driver). The Sparkcontext will be initialized in driver.

2. Wait for Sparkcontext initialization to complete, The maximum number of spark.yarn.applicationMaster.waitTries to wait (default is 10), the program exits if the number of waits is exceeded, otherwise the yarnallocator is initialized with Sparkcontext.

3. When initialization of Sparkcontext and driver is complete, register applicationmaster through Applicationmasterclient to ResourceManager.

4. Assign and start executeors. Before starting executeors, get numexecutors container through yarnallocator and start container in Executeors. (startup Executeors is implemented through executorrunnable, and executorrunnable internal is the boot coarsegrainedexecutorbackend)

5. Finally, the task will run in Coarsegrainedexecutorbackend, and then the health will notify Coarsegrainedscheduler through Akka until the job runs.

Spark on yarn only needs to deploy a spark, and when the application starts, spark registers the associated jar package upload to Resouremanager, the execution of the task is dispatched by ResourceManager, and the code of the spark executes.

Spark on Yarn Architecture parsing