Scheduling and isolation of memory and CPU resources in HadoopYARN

HadoopYARN supports both memory and CPU scheduling (by default, only memory is supported. If you want to further schedule the CPU, You need to configure it yourself ), this article describes how YARN schedules and isolates these resources. In YARN, resource management is completed by ResourceManager and NodeManager.

Hadoop YARN supports both memory and CPU scheduling (only memory is supported by default. If you want to further schedule the CPU, You need to configure it yourself ), this article describes how YARN schedules and isolates these resources. In YARN, resource management is completed by ResourceManager and NodeManager.

Hadoop YARN supports both memory and CPU scheduling (only memory is supported by default. If you want to further schedule the CPU, You need to configure it yourself ), this article describes how YARN schedules and isolates these resources.
In YARN, resource management is completed by ResourceManager and NodeManager. The scheduler in ResourceManager is responsible for resource allocation, while NodeManager is responsible for resource supply and isolation. After ResourceManager allocates resources on a NodeManager to a task (this is called "resource scheduling"), NodeManager needs to provide resources for the task as required, and even ensure that these resources are exclusively occupied, it provides basic guarantee for task running. This is called resource isolation.
For details about the Hadoop YARN Resource scheduler, refer to my article: YARN/MRv2 Resource Manager in-depth analysis-Resource scheduler.
Before officially introducing specific resource scheduling and isolation, let's take a look at the features of memory and CPU resources, which are two different types of resources. The number of memory resources determines the task's life and death. If the memory is not enough, the task may fail to run. In contrast, the CPU resources are different. It only determines the speed of the task, does not affect life and death.
[Scheduling and isolation of memory resources in YARN]
Based on the above considerations, YARN allows users to configure the available physical memory resources on each node. Note that this is "available" because the memory on one node is shared by several services, for example, if you want to configure YARN, HDFS, and HBase, YARN can only be used by you. The configuration parameters are as follows:
(1) yarn. nodemanager. resource. memory-mb
Indicates the total amount of physical memory that YARN can use on the node. The default value is 8192 (MB). Note that if your node's memory resources are not 8 GB, You need to reduce the value, YARN does not intelligently detect the total physical memory of nodes.
(2) yarn. nodemanager. vmem-pmem-ratio
Each task uses 1 MB of physical memory. The maximum amount of virtual memory can be used. The default value is 2.1.
(3) yarn. nodemanager. pmem-check-enabled
Whether to start a thread to check the amount of physical memory being used by each task. If the task exceeds the allocation value, it is killed directly. The default value is true.
(4) yarn. nodemanager. vmem-check-enabled
Whether to start a thread to check the virtual memory used by each task. If the task exceeds the allocated value, it is killed directly. The default value is true.
(5) yarn. scheduler. minimum-allocation-mb
The minimum amount of physical memory that can be applied for by a single task. The default value is 1024 (MB). If the amount of physical memory requested by a task is less than this value, the corresponding value is changed to this value.
(6) yarn. scheduler. maximum-allocation-mb
Maximum physical memory size that can be applied for by a single task. The default value is 8192 (MB ).
By default, YARN uses the thread monitoring method to determine whether the task has used excessive memory. Once the excessive memory usage is found, it is directly killed. Cgroups lacks flexibility in memory control (that is, the task cannot exceed the memory limit at any time. If the limit is exceeded, kill the task directly or report OOM ), when a Java Process is created, the memory doubles and then drops to the normal value. In this case, the thread monitoring method is more flexible (when the process Tree Memory doubles instantly beyond the set value, it can be considered as a normal phenomenon and won't kill the task). Therefore, YARN does not provide the Cgroups memory isolation mechanism.
[Scheduling and isolation of CPU resources in YARN]
In YARN, the CPU resources are still being organized. Currently (version 2.2.0) is only a preliminary implementation method with very coarse granularity. More fine-grained CPU division methods have been proposed, being improved and implemented.
The current CPU is divided into virtual CPU (virtual Core). The virtual CPU here is the concept introduced by YARN itself. The original intention is that considering the CPU performance of different nodes may be different, the computing power of each CPU is also different. For example, the computing power of a physical CPU may be twice that of another physical CPU, you can make up for this difference by configuring multiple virtual CPUs for the first physical CPU. When submitting a job, you can specify the number of virtual CPUs required for each job. In YARN, the CPU configuration parameters are as follows:
(1) yarn. nodemanager. resource. cpu-vcores
The number of virtual CPUs that YARN can use on the node. The default value is 8. Note that we recommend that you set this value to the same number of physical CPU cores. If the number of CPU cores on your node is less than 8, you need to reduce this value. YARN does not intelligently detect the total number of physical CPUs on the node.
(2) yarn. schedres. minimum-allocation-vcores
The minimum number of virtual CPUs that can be applied for by a single task. The default value is 1. If the number of CPUs requested by a task is less than this value, the corresponding value is changed to this number.
(3) yarn. schedres. maximum-allocation-vcores
Maximum number of virtual CPUs that can be applied for by a single task. The default value is 32.
By default, YARN does not schedule CPU resources. You need to configure the corresponding resource Scheduler for your support. For details, refer to these two articles:
(1) Analysis of Hadoop YARN configuration parameters (4)-Fair Scheduler related parameters
(2) Analysis of Hadoop YARN configuration parameters (5)-Capacity schedity related parameters
By default, NodeManager does not isolate CPU resources. You can enable Cgroups to enable CPU isolation.
Due to the uniqueness of CPU resources, the current CPU allocation method is still coarse-grained. For example, many tasks may be IO-intensive and consume very few CPU resources. If you allocate a CPU for it at this time, it is a serious waste, you can make him share a CPU with several other tasks. That is to say, we need to support a more granular CPU expression.
Based on the division of CPU resources in Amazon EC2, The Minimum CPU Unit is EC2 Compute Unit (ECU ), an ECU represents the processing capacity equivalent to 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processors. YARN proposes the Minimum CPU Unit YARN Compute Unit (YCU). Currently, this number is an integer. The default value is 720. nodemanager. resource. cpu-ycus-per-core indicates the computing power of a CPU core (this feature does not exist in version 2.2.0 and may be added to version 2.3.0, when submitting a job, you can directly specify the required YCU. For example, if the value is 360, 1/2 CPU cores are used. The actual performance is that only 1/2 of the computing time of one CPU core is used. Note: At the operating system layer, CPU resources are allocated by time slice. You can say that a process uses 1/3 CPU time slice or 1/5 time slice. For more information about CPU resource division and scheduling, see the following links:

Https://issues.apache.org/jira/browse/YARN-1089

Https://issues.apache.org/jira/browse/YARN-1024

Hadoop new features, improvements, optimizations, and Bug Analysis Series 5: YARN-3
[Summary]
Currently, YARN memory resource scheduling draws on the Hadoop 1.0 method, which is reasonable, but the scheduling method of CPU resources is still being improved. Currently, it is only a preliminary rough implementation, we believe that in the near future, the scheduling of CPU resources in YARN will be improved.