Resource management framework in Hadoop 2.0-YARN (yet another Resource negotiator)

1. Resource management http://dongxicheng.org/mapreduce-nextgen/hadoop-1-and-2-resource-manage/in Hadoop 2.0

Hadoop 2.0 refers to the version of the Apache Hadoop 0.23.x, 2.x or CDH4 series of Hadoop, the core consists of HDFs, mapreduce and yarn three systems, wherein yarn is a resource management system, In charge of cluster resource management and scheduling, MapReduce is the offline processing framework running on yarn, which is the same as the MapReduce in Hadoop 1.0 in the programming model (old and new API) and data processing engine (Maptask and Reducetask) two.

Let us return to the essence of resource allocation, that is, according to the requirements of the task resource allocation of resources in the system. In the real system, the resource itself is multidimensional, including CPU, memory, network I/O and disk I/O, so, if you want to control the allocation of resources accurately, there is no concept of the slot, the most direct way is to have the task directly to the scheduler to request their own resources (such as a task can request 1.5GB Memory and a CPU), and the scheduler allocates the corresponding amount of resources according to the actual requirements of the task, and no longer simply assigns a slot to it, Hadoop 2.0 formally uses this resource allocation scheme based on real resources.

Hadoop 2.0 allows each node (NodeManager) To configure the total amount of available CPU and memory resources, and the central scheduler allocates it to the application based on the total amount of these resources. The node (nodemanager) configuration parameters are as follows:

(1) YARN.NODEMANAGER.RESOURCE.MEMORY-MB

The total amount of physical memory that can be allocated, by default, 8*1024, or 8GB.

(2) Yarn.nodemanager.vmem-pmem-ratio

The task uses the amount of unit physical memory to correspond to the maximum amount of virtual memory that can be used, and the default value is 2.1, which means that each 1MB of physical RAM uses up to 2.1MB of total virtual RAM.

(3) Yarn.nodemanager.resource.cpu-vcore

The number of virtual CPUs that can be allocated, by default, is 8. For finer granularity of CPU resources and CPU performance heterogeneity, yarn allows an administrator to divide each physical CPU into several virtual CPUs based on actual needs and CPU performance, while each administrator can configure the number of available virtual CPUs for each node individually, and when the user submits the application, You can also specify the number of virtual CPUs that are required for each task. For example, there are 8 cpu,node2 on the Node1 node with 16 CPUs, and Node1 CPU performance is node2 twice times, then the two nodes can be configured with the same number of virtual CPUs, such as 32, because the user set the number of virtual CPUs must be an integer, Each task uses at least half the CPU of node2 (not less).

In addition, Hadoop 2.0 introduces a lightweight, cgroups-based resource isolation scheme that significantly reduces the interaction between tasks on the same node, and Hadoop 1.0 uses only JVM-based resource isolation, which is very coarse grained.

Although the resource management scenarios in Hadoop 2 seem perfect, there are a few issues that remain:

(1) The total amount of resources is still statically configured and cannot be modified dynamically. This is already in the perfect, concrete can refer to:

https://issues.apache.org/jira/browse/YARN-291

(2) The CPU is set by the introduction of the "Virtual CPU", while the concept of virtual CPU is vague and ambiguous, and the community is trying to use the ECU concept in Amazon EC2 to regularize it, for specific reference:

https://issues.apache.org/jira/browse/YARN-1024

https://issues.apache.org/jira/browse/YARN-972

(3) can not support group-based resource applications , such as the application of a set of resources to meet certain requirements, the community is also added, specific reference:

https://issues.apache.org/jira/browse/YARN-624

(4) scheduling semantics is not perfect , such as the current application can only request the same priority on the same node of the resource type must be unique, such as from the node Node1 Priority 3 of the resource size is <1 vcore 2048 mb> Otherwise it will be overwritten. At present the community is improving, specific reference:

https://issues.apache.org/jira/browse/YARN-314

Therefore, in the area of resource management, Hadoop 2:1 Advanced, it abandoned the slot-based resource management scheme, using real resource-based management scheme, which will be in resource utilization, resource control, resource isolation and other aspects of the significant improvement, with the Hadoop 2.0 scheduling semantics rich and perfect , it will play a more and more important role.

2. Yarn frame principle and operating mechanism 2.1 yarn frame

(1) ResourceManager: hereinafter referred to as RM. Yarn of the central control module, responsible for the unified planning resources use. It receives a report from NM, builds am, and sends resources to AM.
(2) NodeManager: hereinafter referred to as MM. The resource node module in yarn is responsible for initiating the management container.
(3) Applicationmaster: hereinafter referred to as AM. Each app in yarn starts an AM, is responsible for requesting resources from RM, requesting NM to start container, and telling container what to do.
(4) Container: Abstraction of the task run environment. It describes a series of information: task run resources (including node, memory, CPU), Task Start command, task run environmentin Hadoop 2.0The fundamental idea of refactoring is to separate the two main functions of MR jobtracker into separate components, two of which areResource Managementand thetask scheduling/monitoring. The newResource Managerglobally manage the allocation of all application computing resources, and each application's applicationmaster is responsible for scheduling and coordinating accordingly. An application is nothing more than a single traditional MapReduce task or a DAG (directed acyclic graph) task. The node Management Server for ResourceManager and each machine can manage the user's processes on that machine and can organize the calculations. In fact, each application's Applicationmaster is a detailed framework library that combines the resources obtained from ResourceManager and NodeManager to work together to run and monitor tasks.
The ResourceManager supports hierarchical application queues that enjoy a certain percentage of the cluster's resources. In a sense it is a purely scheduler that does not monitor and state the application during execution. Similarly, it cannot restart tasks that failed because of an application failure or a hardware error. ResourceManager is based on the application of resource demand scheduling; Each application requires a different type of resource, so a different container is required. Resources include: Memory, CPU, disk, network and so on. As you can see, there is a significant difference between this and the existing Mapreduce fixed type Resource usage model, which has a negative impact on the use of the cluster. The resource Manager provides a plug-in for scheduling policies that is responsible for assigning cluster resources to multiple queues and applications. Scheduling plug-ins can be based on existing capacity scheduling and fair scheduling models. NodeManager is the agent for each machine framework, a container for executing applications, monitoring the application's resource usage (CPU, memory, hard disk, network) and reporting to the scheduler.
The applicationmaster responsibilities of each application are to ask the scheduler for the appropriate resource containers, to run tasks, to track the status of the application and to monitor their processes, and to handle the reasons for the failure of the task. The advantages of the 2.2 Yarn frame relative to the old MapReduce framework are 1. This design greatly reduces the resource consumption of Jobtracker (now the ResourceManager) and makes it more secure and graceful to distribute the programs that monitor the status of each job subtask (tasks).
2. In the new Yarn, Applicationmaster is a changeable part that allows users to write their own appmst on different programming models, allowing more types of programming models to run in the Hadoop cluster, referring to the map in the Hadoop Yarn official configuration template Red-site.xml configuration.
3. The representation of the resource in memory (in the current version of Yarn does not take into account the CPU footprint) is more reasonable than the number of slots remaining.
4. In the old framework, jobtracker a big burden is to monitor the operation of tasks under the job, and now, this part is thrown to Applicationmaster, and ResourceManager has a module called application Smasters (note not applicationmaster), which monitors the health of Applicationmaster and restarts it on other machines if a problem occurs.
5. Container is a framework presented by Yarn for future resource isolation. This should draw on the work of Mesos, is currently a framework, only to provide the isolation of Java Virtual machine memory, Hadoop team design ideas should be able to support more resource scheduling and control, since the resources expressed as the amount of memory, then there is no previous map slot/reduce slot The embarrassing situation that separates the cluster resources from idle. 2.3 Yarn-based Hadoop 2.0 ecosystem a new generation of yarn container frameworks is an upgraded version of the traditional Mr Hadoop container framework, and the previous Mr Deployment architecture relies on jobtracker and tasktracker interaction patterns, while the next generation of yarn The container framework, with ResourceManager and NodeManager interaction patterns, a higher level of abstraction and architectural design, is the Yarn container framework capable of supporting a wide range of computing engines, including traditional Hadoop Mr and now newer spark. The resource management framework in Hadoop 2.0, which is a framework manager that allocates resources and provides a runtime environment for various frameworks. While MRv2 is the first computational framework running on yarn, other computational frameworks, such as Spark, Storm, are being ported to yarn. (1) Off-line computing framework: MapReduce (2) Dag Computing Framework: Tez (3) Streaming computing framework: Storm (4) Memory Computing Framework: Spark (5) Figure Calculation Framework: Giraph,graphlib Note: All of the above is from the Internet.

Resource Management framework in Hadoop 2.0-YARN (yet another Resource negotiator)