Preliminary understanding of yarn architecture and principles

1. What is yarn?

From the changes in the use of Distributed Systems in the industry and the long-term development of the hadoop framework, the jobtracker/tasktracker mechanism of mapreduce needs to be adjusted in a large scale to fix its scalability, memory consumption, and thread model, defects in reliability and performance. In the past few years, the hadoop development team has fixed some bugs, but the costs of these fixes are getting higher and higher, which indicates the difficulty of making changes to the original framework. To fundamentally solve the performance bottleneck of the old mapreduce framework and promote the long-term development of the hadoop framework, hadoop's mapreduce framework has undergone a fundamental change since version 0.23.0. The new hadoop mapreduce framework is named mapreducev2 or yarn.

Yarn is a new resource management system introduced from version 0.23.0 and evolved directly from MR1 (0.20.x, 0.21.x, 0.22.x). Its core idea is as follows:

The Resource Management and job call functions of jobtracker in MR1 are separated by the ResourceManager and applicationmaster processes.

1) ResourceManager: Manages and schedules resources of the entire cluster.

2) applicationmaster: Responsible for application-related transactions, such as task scheduling, task monitoring, and fault tolerance.

2. Why is yarn used?

Compared with the old mapreduce, yarn adopts a hierarchical cluster framework, which solves a series of defects in the old mapreduce and has the following advantages.

1. HDFS Federation is proposed, which allows multiple namenode to manage different directories for access isolation and horizontal scaling. The namenode Hot Standby solution (namenode ha) is used for single point of failure in the running namenode.

2. yarn splits Resource Management and Application Management into two parts, namely, resoucemanager and applicationmaster. resoucemanager manages resource management and scheduling, applicationmaster is responsible for task splitting, task scheduling, and fault tolerance related to specific applications. Each application corresponds to an applicationmaster.

3. yarn is backward compatible. Jobs run on mrv1 can run on Yarn without any modifications.

4. The resource representation is in memory (in the current yarn version, CPU usage is not taken into account), which is more reasonable than the number of remaining slots.

5. supports multiple frameworks. yarn is no longer a simple computing framework, but a framework manager. You can Port various computing frameworks to yarn, yarn is used for unified management and resource allocation. Currently, multiple computing frameworks can be run on yarn, such as mapreduce, storm, spark, and flink.

6. it is easier to upgrade the framework. In yarn, various computing frameworks are no longer deployed as a service to each node of the cluster (for example, the mapreduce framework does not need to deploy services such as jobtracler and tasktracker ), instead, it is encapsulated into a user Library (LIB) and stored on the client. to upgrade the computing framework, you only need to upgrade the user library. How easy is it!

3. What is the composition of the yarn architecture?

First, let's take a look at the yarn architecture diagram, as shown in.

Yarn consists of the following components: ResourceManager, nodemanager, applicationmaster, and container.

1. ResourceManager (RM)

The essence of the yarn hierarchy is ResourceManager. This entity controls the entire cluster and manages the allocation of applications to basic computing resources. ResourceManager carefully arranges various resource components (such as computing, memory, and bandwidth) to the basic nodemanager (each node proxy of yarn ). ResourceManager also allocates resources with applicationmaster to start and monitor their basic applications together with nodemanager. In this context, applicationmaster assumes some of the previous tasktracker roles, and ResourceManager assumes the jobtracker role.

In general, RM has the following functions:

1) process client requests

2) Start or monitor applicationmaster

3) monitor nodemanager

4) resource allocation and Scheduling

2. applicationmaster (AM)

Applicationmaster manages each application instance running in yarn. The applicationmaster is responsible for coordinating resources from ResourceManager and monitoring container execution and resource usage (Resource Allocation such as CPU and memory) through nodemanager ). Please note that, although the current resources are more traditional (CPU core, memory), there will be new resource types based on the task at hand (such as specific processing units or dedicated processing devices) in the future ). From the perspective of yarn, applicationmaster is a user code, so there are potential security problems. Yarn assumes that the applicationmaster has errors or is even malicious, so it treats them as unprivileged code.

In general, am has the following functions:

1) responsible for data splitting

2) apply for resources for applications and assign internal tasks

3) task monitoring and fault tolerance

3. nodemanager (Nm)

Nodemanager manages each node in the yarn cluster. Nodemanager provides services for each node in the cluster, From supervising the lifetime management of a container to monitoring resources and tracking node health. Mrv1 manages the execution of MAP and reduce tasks through slots, while nodemanager manages abstract containers, which represent resources available for a specific application for each node.

In general, nm has the following functions:

1) manage resources on a Single Node

2) process commands from ResourceManager

3) process commands from applicationmaster

4. Container

Container is the resource abstraction in yarn. It encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, and network. When am applies for resources from RM, rm is the resource returned by AM in the container format. Yarn assigns a container to each task, and the task can only use the resources described in the container.

In general, container has the following functions:

1) Abstract The task running environment, encapsulate resources, environment variables, startup commands, and other task running information in multiple dimensions such as CPU and memory.

 

To use a yarn cluster, you must first request a customer that contains the application. ResourceManager negotiates necessary resources of a container and starts an applicationmaster to indicate submitted applications. By using a resource request protocol, applicationmaster negotiates the resource containers on each node for the application. When the application is executed, applicationmaster monitors the container until it is complete. When the application is completed, applicationmaster logs out of its container from ResourceManager and the execution cycle is complete.

 

Through the above explanation, it should be clear that the old hadoop architecture is highly constrained by jobtracker, and jobtracker is responsible for the resource management and Job Scheduling of the entire cluster. The new yarn architecture breaks this model and allows a new ResourceManager to manage resource usage across applications. applicationmaster is responsible for managing job execution. This change eliminates a bottleneck and improves the ability to extend a hadoop cluster to a much larger configuration than before. In addition, unlike traditional mapreduce, yarn allows standard communication modes such as message passing interface (MPI) to run different programming models at the same time, it includes graphic processing, iterative processing, machine learning, and general cluster computing.

4. Yarn Principle

Yarn jobs are mainly composed of the following steps:

1. Submit a job

The client calls the job. waitforcompletion method and submits a mapreduce job to the entire cluster (step 1 ). The new job ID (Application ID) is allocated by the Resource Manager (step 1 ). the client of the job verifies the output of the job, computes the split of input, and copies the job resources (including the jar package, configuration file, and split information) to HDFS (step 1 ). finally, submit the job by calling Resource Manager submitapplication () (step 1 ).

2. Job Initialization

When the resource manager receives a submitapplciation () request, it sends the request to the scheduler (schediner), The scheduler allocates the iner, and then the resource manager starts the Application Manager Process in the container, monitored by the Node Manager (step 1 ).

The Application Manager of mapreduce jobs is a Java application whose main class is mrappmaster. it creates some bookkeeping objects to monitor the job progress and obtain the task progress and completion report (step 1 ). then, the distributed file system is used to obtain the input split calculated by the client (step 1 ). create a map task for each input split according to mapreduce. job. reduces creates a reduce task object.

3. Task Allocation

If the job is small, the application manager will choose to run the task in its own JVM.

If it is not a small job, the application manager Requests the container from the resource manager to run all MAP and reduce tasks (step 1 ). these requests are transmitted through heartbeat, including the data location of each map task, such as the host name and rack (Rack) of the input split ). the scheduler uses this information to schedule tasks. It tries its best to allocate the tasks to the nodes that store data, or to the nodes that store the same racks as the nodes that input the split.

4. task running

After a task is assigned to a container by the scheduler of the resource manager, the application manager starts the container (step 1) by contacting the Node Manager ). A task is executed by a Java application whose main class is yarnchild. before running a task, you must first localize the resources required by the task, such as job configuration, Jar files, and all distributed cache files (step 1 ). finally, run the map or reduce task (step 1 ).

Yarnchild runs in a dedicated JVM, but yarn does not support JVM reuse.

5. Update progress and status

The job in yarn returns its progress and status (including counter) to the application manager, and the client returns the progress and status (through mapreduce. client. progressmonitor. pollinterval settings) request progress updates to the Application Manager and display them to users.

6. job completed

In addition to requesting the job progress from the application manager, the client will call waitforcompletion () every five minutes to check whether the job is complete. the time interval can be passed through mapreduce. client. completion. set pollinterval. after the job is completed, the application manager and container clean up the job status, and the outputcommiter job cleanup method is also called. the job information is stored by the job history server for user verification.

 

Http://www.cnblogs.com/codeOfLife/p/5492740.html

 

Preliminary understanding of yarn architecture and principles (conversion)