Hadoop YARN FAQs and Solutions

This article summarizes several hadoop yarn in http://www.aliyun.com/zixun/aggregation/17253.html "> common problems and solutions, note that this article describes the solution for hadoop 2.2.0 and above.

(1) By default, the load of each node is not balanced (the number of tasks is different). Some nodes run many tasks and some do not have tasks. How to make each node tasks as balanced as possible?

A: By default, the resource scheduler is in batch scheduling mode, that is, a heartbeat will allocate as many tasks as possible. In this way, a node sending a heartbeat will take the task as its priority (premise: the number of tasks is much smaller than that of a cluster at the same time Run the number of tasks), in order to avoid this from happening, you can configure the parameters as follows:

If you are using a fair scheduler, set the parameter yarn.scheduler.fair.max.assign to 1 in yarn-site.xml (the default is -1)

If you use the capacity scheduler (default scheduler), can not be configured, the scheduler does not currently have such functions as load balancing.

Of course, from the Hadoop cluster utilization point of view, this problem is not a problem, because in general, the number of user tasks is much larger than the cluster's concurrent processing capacity, that is, usually, the cluster is always busy, there is no Node has been idle.

(2) too many tasks on a node, resource utilization is too high, how to control the number of tasks on a node?

A: The number of tasks running on a node is mainly determined by two factors, one is the total amount of resources available to the NodeManager, and the other is the resource requirements for a single task. For example, a resource available on a NodeManager is 8 GB of memory, 8 cpu, a single The task resource requirement is 1 GB memory, 1 cpu, then the node runs up to 8 tasks.

Resources available on the NodeManager are configured by the administrator in the configuration file yarn-site.xml with the following parameters:

yarn.nodemanager.resource.memory-mb: The total amount of available physical memory, the default is 8096

yarn.nodemanager.resource.cpu-vcores: The total number of available CPUs, the default is 8

For MapReduce, the amount of task resources per job can be set with the following parameters:

mapreduce.map.memory.mb: physical memory, the default is 1024

mapreduce.map.cpu.vcores: CPU number, the default is 1

Note: A detailed description of these configuration attributes can be found in the article: Analysis of Hadoop YARN Configuration Parameters (1) -RM and NM related parameters.

By default, each scheduler will only be scheduled on the memory resources, CPU resources will not be considered, you need to set the scheduler configuration file, the specific reference article: Analysis of Hadoop YARN configuration parameters (4) -Fair Scheduler related parameters And Hadoop YARN configuration parameter analysis (5) -Capacity Scheduler related parameters.

(3) how to set the amount of memory occupied by a single task and the number of CPU?

A: For MapReduce, the amount of task resources for each job can be set with the following parameters:

mapreduce.map.memory.mb: physical memory, the default is 1024

mapreduce.map.cpu.vcores: CPU number, the default is 1

It should be noted that, by default, each scheduler will only memory resources scheduling, CPU resources will not be considered, you need to be in the scheduler configuration file settings.

(4) The amount of memory the user has set for the task is 1000MB, why the memory allocated eventually is 1024MB?

A: In order to manage resources and schedule resources easily, Hadoop YARN has a built-in resource normalization algorithm that specifies the minimum amount of requisite resources, the maximum amount of resources that can be applied and the resource regularization factor. If the amount of resources requested by an application is less than the minimum amount that can be applied Resources, then YARN will be the size of the minimum amount of application, that is, the application of resources will not be less than the resources they apply, but not necessarily equal; if the application resources is greater than the maximum amount of resources Amount will throw an exception and can not apply for success. The normalization factor is used to normalize the application resource. If the resource requested by the application program is not an integer multiple of the factor, the normalized factor is modified to a value corresponding to the smallest integer multiple , The formula is ceil (a / b) * b, where a is the resource applied by the application and b is the normalization factor.

The parameters described above need to be set in yarn-site.xml, the relevant parameters are as follows:

yarn.scheduler.minimum-allocation-mb: The minimum amount of memory available, the default is 1024

yarn.scheduler.minimum-allocation-vcores: The minimum number of CPUs that can be applied, the default is 1

yarn.scheduler.maximum-allocation-mb: The maximum amount of memory that can be applied, the default is 8096

yarn.scheduler.maximum-allocation-vcores: The maximum number of CPUs that can be applied, the default is 4

For the normalization factor, different schedulers are different, as follows:

FIFO and Capacity Scheduler, the normalization factor equal to the minimum amount of resources that can be applied, can not be configured separately.

Fair Scheduler: The normalization factor is set by the parameter yarn.scheduler.increment-allocation-mb and yarn.scheduler.increment-allocation-vcores. The default is 1024 and 1.

It can be seen from the above introduction that the amount of resources applied by the application to the resource may be greater than the amount of resources requested by the resource. For example, the minimum amount of resources that YARN can apply for is 1024 and the normalization factor is 1024. If an application requests 1500 memory, If the rounding factor is 512, you get 1536 memory.

(5) We are using Fairscheduler, configured with multiple queues, when the user submits a job, the specified queue does not exist, Fair Scheduler will automatically create a new queue instead of an error (such as error: queue XXX does not exist), how Avoid this happening?

A: Set yarn.scheduler.fair.allow-undeclared-pools in yarn-site.xml and its value to false (the default is true).

(6) Hadoop 2.0 process, encountered an error, how to troubleshoot errors?

A: Starting from hadoop log, Hadoop log storage location can refer to my article: Hadoop log in the end where?