Introduction to the capacity scheduler of hadoop 0.23 (hadoop mapreduce next generation-capacity schedity)

Original article: http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

This document describes capacityscheduler, a pluggable hadoop scheduler that allows multiple users to securely share a large cluster, their applications can obtain the required resources within the capacity limit.

Overview

Capacityscheduler is designed to enable hadoop applications to run on clusters with shared, multi-user, and ease of operation, while maximizing the cluster throughput and utilization.

Traditionally, each organization has its own set of machine resources to ensure that the peak volume is close to the peak, which usually leads to poor resource utilization and management overhead of multiple independent clusters, sharing clusters between organizations is a very beneficial method. However, organizations are worried about sharing clusters and their resources are occupied, which affects SLA.

Capacityscheduler is designed to share a large cluster while ensuring the capacity requirements of each organization. His core idea is to use various hadoop Cluster organizations to build a common fund and share resources based on their resource requirements. This method has an additional benefit, an organization can access resources that are not used by others, thus providing a cost-effective flexible organization.

Cross-organization cluster sharing requires the strong support of users in multiple organizations, because each organization must ensure capacity and security to ensure that the shared cluster is not affected by a single application or user. Capacityscheduler provides a set of strict limits to ensure that a single application or user does not consume an inappropriate proportion of cluster resources. In addition, capacityscheduler provides initialization/suspension of a user or queue application to ensure fairness and Stability of the cluster.

A major abstraction provided by capacityscheduler is queue, which is created by an administrator to reflect resource allocation.

In the future, resources will be further controlled and predicted. capacityschedu provides a hierarchical queue to ensure that resources can obtain idle resources in the sub-queue of a queue before other queues, in this way, resources are preferentially shared within an organization.

Features

Capacsche supports the following features:

• Hierarchical queue

Hierarchical queues support the priority to share resources in the subqueues of an organization, thus providing more control and prediction capabilities.

• Resource Capacity guarantee

From a certain point of view, the queue implements a resource division, and all applications are designated to a specific queue. The resources that these applications can use are limited by the resources owned by the queue, the administrator can configure soft limits or optional Hard Limits to limit the resources owned by the queue.

• Security

Each queue has a strict ACL (accesscontrol List) to control which users can access the queue, and there is a safe-guard to ensure that users cannot view or modify applications of other users, in addition, you can set the administrator role for each queue or system.

• Elasticity

Idle resources can be allocated to any queue, which may exceed the queue resource limit. That is to say, if the cluster has idle resources, and the resources required by some queues exceed the limit assigned to them, these idle resources will be allocated to these queues, this ensures the predictability and elasticity of resources, thus preventing the island of labor and helping to optimize the utilization of resources.

Multi-user, a series of comprehensive settings can prevent a single application or users from occupying all the resources of the queue or cluster, prevent the cluster from being used excessively by a single user, thus ensuring that multiple users can use the cluster together

• Operability

Runtimeconfiguration: some settings can be configured at runtime, such as the resource allocation capacity and ACL (Access Control List). These settings can be set by the Administrator at runtime to reduce the impact on users. In addition, a terminal is provided to the Administrator and user to view the resources of each queue currently allocated. The administrator can also add a queue at runtime.

Drain applications: drainapplications is a queue that administrators can stop running. It also ensures that the tasks in the queue are completed and new tasks are not submitted to the queue. If a queue is in the stopped status, new applications will not be submitted to the queue or its sub-queue, and existing tasks in the queue will continue to run. The administrator can also start a stoped queue.

• Resource-Based Scheduling

Supports resource-intensive applications. Applications can be allocated more resources than the default settings. Therefore, they can accommodate applications with different resource requirements. Currently, only the configuration of memory resources is supported.

Configuration

Set ResourceManager to use capacityscheduler

To enable ResourceManager to use capacityschedager, you need to set the attributes below the conf/yan-set.xml

Property	Value
Yarn. ResourceManager. schedager. Class	Org. Apache. hadoop. yarn. server. ResourceManager. schedager. FIFO. capacityschedager

Create a queue

Conf/capacity-schduler.xml is the configuration file for capacityscheduler

Capacityscheduler has a predefined queue named root. All queues in the system are root subqueues.

In addition, the queue can be created by configuring the Yan. schedity. Capacity. Root. Queues configuration item, and the sub-queue list can be created by commas (,).

Capacityscheduler uses the queue path concept to establish a hierarchical sub-queue. The queue path is a full path from the root to the sub-queue, which is separated ".".

A defined sub-queue can be used as a reference for further configuration of knob, for example, Yan. scheduler. capacity. <queue-path>. queues

Below are three top-level sub-queues A, B, C, and A, B.

<Property>

<Name> yarn. schedity. Capacity. Root. Queues </Name>

<Value> A, B, C </value>

<Description> the queues at the thislevel (root is the root Queue ).

</Description>

</Property>

 

<Property>

<Name> yarn. schedity. Capacity. Root. A. Queues </Name>

<Value> A1, A2 </value>

<Description> the queues at the thislevel (root is the root Queue ).

</Description>

</Property>

 

<Property>

<Name> yarn. schedity. Capacity. Root. B. Queues </Name>

<Value> B1, B2, B3 </value>

<Description> the queues at the thislevel (root is the root Queue ).

</Description>

</Property>

Queue attributes

Resource allocation

Property	Description
Yarn. schedity. Capacity. <queue-path>. Capacity	QueueCapacityIn percentage (% ). the sum of capacities for all queues, at each level, shocould be less than or equal to 100. applications in the queue may consume more resources than the queue's capacity if there are free resources, providing elasticity.
Yarn. schedity. Capacity. <queue-path>. Maximum-capacity	Maximum queue capacity in percentage (%). This limitsElasticityFor applications in the queue.
Yarn. schedity. Capacity. <queue-path>. Minimum-user-limit-percent	Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. the user limit can vary between a minimum and maximum value. the former depends on the number of users who have submitted applications, And the latter is set to this property value. for e.g ., suppose the value of this property is 25. if two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. if a third user submits an application, no single User can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed.
Yarn. schedity. Capacity. <queue-path>. User-limit-Factor	The multiple of the queue capacity which can be configured to allow a single user to acquire more resources. by default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle th Cluster Is.

Restrictions on running and pending applications

Capacityscheduler supports the following parameters to control running and suspended applications.

Property	Description
Yarn. schedity. Capacity. Maximum-Applications	Maximum number of jobs in the system which can be concurently active both running and pending. Limits on each queue are directly proportional to their queue capacities.
Yarn. schedity. Capacity. Maximum-am-resource-percent	Maximum percent of resources in the cluster which can be used to run application masters-controls number of concurrent running applications.

Queue administration and permissions

Capacityscheduler supports the following parameters to manage queues.

Property	Description
Yarn. schedity. Capacity. <queue-path>. State	TheStateOf the queue. can be oneRunningOrStopped. If a queue is inStoppedState, new applications cannot be submittedItselfOrAny of its child queueus. Thus, ifRootQueue isStoppedNo Applications can be submitted to the entire cluster. Existing applications continue to completion, thus the queue can beDrainedGracefully.
Yarn. schedity. Capacity. Root. <queue-path>. acl_submit_jobs	TheACLWhich controls who canSubmitJobs to the given queue. If the given user/group has necessary ACLs on the given queue orOne of the parent queues in the hierarchyThey can submit jobs.
Yarn. schedity. Capacity. Root. <queue-path>. acl_administer_jobs	TheACLWhich controls who canAdministerJobs on the given queue. If the given user/group has necessary ACLs on the given queue orOne of the parent queues in the hierarchyThey can administer jobs.

Reviewcapacityscheduler Configuration

Once the installation and configuration are complete, you can start yarn from web-UI:

• Start yarn normal
• Open ResourceManager webui
• The scheduler web-page displays the resource usage of each individualqueue.

Change the queue Configuration

Changing opposite attributes and adding new queues is very easy, you need to edit CONF/capacity-scheduler.xml and then run rmadmin-refreshqueue

$ vi $HADOOP_CONF_DIR/capacity-scheduler.xml

$ $YARN_HOME/bin/rmadmin -refreshQueues

Note: The queue cannot be deleted. Only New queues are supported. The update queue configuration is valid and the queue capacity is equal to 100% at each level. (Translator: I don't know how to understand this sentence. The original Article is as follows:Note:Queues cannot beDeleted, Only addition of newqueues is supported-the updated queue configuration shoshould be a valid onei. E. queue-capacity at eachLevelShocould
Be equal to 100% .)