Hadoop Yarn Scheduler

Hadoop Yarn Scheduler

Ideally, our application requests to Yarn resources should be met immediately, but in reality resources are often limited, especially in a very busy cluster, requests for an application resource often need to wait for a period of time to get to the corresponding resource. In Yarn, Scheduler is used to allocate resources to applications. In fact, scheduling is a difficult problem. It is difficult to find a perfect strategy to solve all application scenarios. To this end, Yarn provides a variety of schedulers and configurable policies for us to choose from.

1. scheduler Selection

There are three schedulers available in Yarn:FIFO Scheduler,Capacity Scheduler,FairScheduler.

FIFO SchedulerQueues applications in the order of submission. This is a first-in-first-out queue. when allocating resources, allocate resources to the applications in the queue first, after the top application requirements are met, the next allocation will be given, and so on.

FIFO SchedulerIt is the simplest and easiest to understand scheduler and does not require any configuration, but it is not applicable to shared clusters. Large applications may occupy all cluster resources, which causes blocking of other applications. More suitable for shared ClustersCapacity SchedulerOrFair SchedulerBoth schedulers allow large and small tasks to obtain certain system resources while submitting them.

Below"Yarn scheduler comparison chart"The differences between these schedulers are displayed. We can see that in the FIFO scheduler, small tasks are blocked by large tasks.

For the Capacity scheduler, a dedicated queue is used to run small tasks. However, setting a dedicated queue for a small task will occupy certain cluster resources in advance, this causes the execution time of large tasks to lag behind the time when the FIFO scheduler is used.

In the Fair scheduler, we do not need to occupy certain system resources in advance. The Fair scheduler dynamically adjusts system resources for all running jobs. As shown in, when the first large job is submitted, only this job is running and it obtains all cluster resources. After the second small task is submitted, the Fair scheduler allocates half of the resources to this small task, so that the two tasks can share the cluster resources fairly.

Note that in the Fair scheduler, there is a certain delay from submitting the second task to obtaining resources, because it needs to wait for the first task to release the occupied Container. After a small task is executed, the resources occupied by the task are released, and the large task receives all the system resources. The final result is that the Fair scheduler achieves high resource utilization and ensures that small tasks can be completed in a timely manner.

Comparison chart of Yarn Scheduler: 

Ii. Capacity schedity (container Scheduler) configuration 2.1 Introduction to container Scheduling

The Capacity scheduler allows multiple organizations to share the entire cluster. Each organization can obtain a portion of the computing power of the cluster. By assigning a dedicated queue to each organization and then Assigning certain cluster resources to each queue, the entire cluster can provide services to multiple organizations by setting multiple queues. In addition, the queue can be vertically divided, so that multiple members in an organization can share the queue resources. Inside a queue, resource Scheduling adopts the first-in-first-out (FIFO) policy.

Through the figure above, we know that a job may not be able to use the resources of the entire queue. However, if the queue runs multiple jobs and the queue has enough resources, it is allocated to these jobs. What if the queue has insufficient resources? In fact, the Capacity scheduler may still allocate additional resources to this queue, which is the concept of "queue elasticity.

In normal operations, the Capacity scheduler does not forcibly release the iner. When a queue has insufficient resources, this queue can only obtain the iner resources released by other queues. Of course, we can set a maximum resource usage for the queue to avoid occupying too many idle resources in this queue, so that other queues cannot use these idle resources, this is the trade-off between "elastic queue.

2.2 configure container Scheduling

Suppose we have queues at the following levels:

root├── prod└── dev    ├── eng    └── science

The following is the configuration file of a simple Capacity scheduler. The file name iscapacity-scheduler.xml. In this configuration, two sub-queues are defined under the root queue.prodAnddev, Accounting for 40% and 60% respectively. Note that the configuration of a queue is based on attributes.yarn.sheduler.capacity.<queue-path>.<sub-property>Specified,<queue-path>Indicates the inheritance tree of the queue, suchroot.prodQueue,<sub-property>GenerallycapacityAndmaximum-capacity.

We can see that,devThe queue is dividedengAndscienceTwo subqueues with the same capacity.devOfmaximum-capacityAttribute is set to 75%, so even ifprodThe queue is completely idle.devIt does not occupy all cluster resources, that is,prodThere are still 25% of available resources in the queue for emergency response. We noticed that,engAndscienceTwo queues are not setmaximum-capacityAttribute, that isengOrscienceThe job in the queue may use the entiredevAll resources in the queue (up to 75% of the cluster ). Similarly,prodBecause the maximum-capacity attribute is not set, it may occupy all cluster resources.

In addition to configuring the queue and its Capacity, the Capacity container can also configure the maximum number of resources that a user or application can allocate, the number of applications that can run simultaneously, and the queue ACL authentication.

2.3 queue settings

The setting of the queue depends on our specific application. For example, in MapReduce, we can usemapreduce.job.queuenameAttribute specifies the queue to use. If the queue does not exist, we will receive an error when submitting the task. If we do not define any queues, all applications will be placed in onedefaultQueue.

NOTE: For the Capacity scheduler, our queue name must be the last part of the queue tree. If we use the queue tree, it will not be recognized. For example, in the above configuration, we useprodAndengYes as the queue name, but if we useroot.dev.engOrdev.engIs invalid.

3. Fair sched (Fair Scheduler) configuration 3.1 Fair Scheduling

The Fair scheduler is designed to allocate Fair resources to all applications (Fair definition can be set through parameters ). Above"Yarn scheduler comparison chart"Shows the fair scheduling of two applications in a queue. Of course, fair scheduling can also work between multiple queues. For example, assume that two users A and B have A queue respectively. When A starts A job and B does not have A job, A obtains all cluster resources. When B starts A job, A's job continues to run, however, after a while, the two tasks will each obtain half of the cluster resources. If B starts the second job again and other jobs are still running, it will share the resources of the queue B with the first job of B, that is, the two jobs of B will be used for 1/4 of cluster resources, while the job of A is still used for half of cluster resources. The result is that the resources will be shared equally between the two users. The process is shown in:

3.2 enable Fair Scheduler

The scheduler usesyarn-site.xmlIn the configuration fileyarn.resourcemanager.scheduler.classIf the parameter is configured, the Capacity Scheduler is used by default. If you want to use the Fair scheduler, You need to configure the full qualified name of the FairScheduler class on this parameter:org.apache.Hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.

3.3 queue Configuration

The configuration file of the Fair scheduler is located infair-scheduler.xmlFile.yarn.scheduler.fair.allocation.fileModify the attributes. If this configuration file is not available, the Fair scheduler uses the allocation policy. This policy is similar to that described in section 3.1: The scheduler automatically creates a queue when the user submits the first application, the queue name is the user name, and all applications are allocated to the corresponding user queue.

We can configure each queue in the configuration file and configure the queue in different layers like the Capacity scheduler. For example, refercapacity-scheduler.xmlTo configure fair-scheduler:

The queue hierarchy is through nesting<queue>Element implementation. All queues arerootChildren in the queue, even if we are not assigned<root>Element. In this configuration, we setdevThe queue is dividedengAndscienceTwo queues.

The queue in the Fair scheduler has a weight attribute (this weight is the definition of fairness) and uses this attribute as the basis for Fair scheduling. In this example, when the scheduler assigns a cluster40:60ResourceprodAnddevIs regarded as fair,engAndscienceIf no weight is defined for the queue, It is evenly allocated. Here, the weight is not a percentage. we replace the 40 and 60 values with 2 and 3 respectively, and the effect is the same. Note: For queues automatically created by user when no configuration file exists, they still have weight and the weight value is 1.

Each queue can still have different scheduling policies. The default scheduling policy of the queue can use top-level elements<defaultQueueSchedulingPolicy>Configuration. If not configured, fair scheduling is used by default.

Although it is a Fair scheduler, it still supports FIFO scheduling at the queue level. The scheduling policy of each queue can be<schedulingPolicy>Element overwrite. In the above example,prodThe queue is specified to use FIFO for scheduling. ThereforeprodThe tasks in the queue can be executed in the FIFO Rule Order. Note that,prodAnddevThe scheduling between them is still fair scheduling, the sameengAndscienceFair scheduling.

Although not shown in the preceding configuration, you can configure the maximum, minimum resource usage, and maximum number of running applications for each queue.

3.4 queue settings

The Fair scheduler uses a rule-based system to determine which queue the application should be placed in. In the preceding example,<queuePlacementPolicy>An element defines a rule list. Each rule is tasted one by one until the matching is successful. For example, the first rule in the previous examplespecifiedThe application is placed in the specified queue. If the application does not specify the queue name or the queue name does not exist, the rule is not matched and the next rule is attempted.primaryGroupThe rule will try to put the application inUnix group name of the userIn the named queue, if this queue is not available, try the next rule instead of creating a queue. If none of the current rules are metdefaultRules, put the application indev.engQueue.

Of course, we do not need to configurequeuePlacementPolicyThe scheduler uses the following rules by default:

<queuePlacementPolicy><rule name="specified" /><rule name="user" /></queuePlacementPolicy>

The above rule can be summarized as one sentence. Unless the queue is accurately defined, the queue will be created with the queue name as the user name.

There is also a simple configuration policy that puts all applications in the same Queue (default), so that all applications can share clusters equally, rather than between users. The configuration is defined as follows:

<queuePlacementPolicy><rule name="default" /></queuePlacementPolicy>

To implement the above functions, we can also directly set it without using the configuration file.yarn.scheduler.fair.user-as-default-queue=falseIn this way, the application will be put into the default queue, rather than the queue of each user name. You can also setyarn.scheduler.fair.allow-undeclared-pools=falseIn this way, the user cannot create a queue.

3.5 Preemption)

When a job is submitted to an empty queue in a busy cluster, the job is not executed immediately, but blocked until the running job releases system resources. To make the execution time of submitted jobs more predictable (you can set the wait timeout time), the Fair scheduler supports preemption.

Preemption is to allow the scheduler to kill the containers that occupy the resource queue that exceeds its expected share. These containers resources can be allocated to the queue that should have these shares of resources. It is important to note that preemption will reduce the execution efficiency of the cluster because the terminated containers needs to be re-executed.

You can set a global Parameteryarn.scheduler.fair.preemption=trueTo enable the preemption function. In addition, there are two parameters to control the expiration time of the preemption (these two parameters are not configured by default, and at least one must be configured to allow the preemption of the Container ):

- minimum share preemption timeout- fair share preemption timeout

Ifminimum share preemption timeoutIf the minimum resource guarantee is not obtained within the specified time, the scheduler will seize containers. We can use top-level elements in the configuration file<defaultMinSharePreemptionTimeout>Configure the timeout time for all queues. We can also<queue>Configuration in element<minSharePreemptionTimeout>To specify the timeout time for a queue.

Similarly, if the queuefair share preemption timeoutIf you do not obtain half of the equal resources within the specified time (this ratio can be configured), The scheduler will seize containers. This timeout time can be passed through the top-level element<defaultFairSharePreemptionTimeout>And element-level elements<fairSharePreemptionTimeout>Configure the time-out time for all queues and a queue respectively. The ratio mentioned above can be<defaultFairSharePreemptionThreshold>(Configure all queues) and<fairSharePreemptionThreshold>(Configure a queue). The default value is 0.5.

Hadoop2.3-HA high-availability cluster environment construction

Hadoop project-Cloudera 5.10.1 (CDH) installation and deployment based on CentOS7

Detailed explanation of Hadoop2.7.2 cluster construction (high availability)

Use Ambari to deploy a Hadoop cluster (build an intranet HDP source)

Hadoop cluster installation in Ubuntu 14.04

Install Hadoop 2.7.2 In CentOS 6.7

Build a distributed Hadoop-2.7.3 cluster on Ubuntu 16.04

Build a Hadoop 2.6.4 distributed Cluster Environment in CentOS 7

Hadoop2.7.3 + Spark2.1.0 fully distributed cluster Construction Process