Hadoop-2.2.0 China document--mapreduce Next Generation--fair dispatch

Purpose

This document describes a pluggable FairScheduler scheduler for Hadoop, agreeing that YARN applications share resources fairly in a large cluster.

Simple Introduction

Fair scheduling is a method of allocating resources to applications.  So that in the end, all applications on average get equal resources. Hadoop NextGen can dispatch multiple types of resources. By default, Fair Scheduler only makes a fair scheduling decision based on memory. The memory and CPU can be dispatched using dominant Resource fairness concepts developed with GHODSI and so on. When only one application executes, the application uses the entire cluster.

After other apps have been submitted, the empty resources are allocated to the new application, so that each application can finally get roughly the same amount of resources.

Unlike the default Hadoop scheduler, which makes the application a queue, this allows short applications to be completed in a reasonable amount of time, while long applications are not hungry.

)。 This is also a reasonable way to share a cluster in multiple users.

Finally, fair sharing can be applied to application priorities--priority is used to determine the weight of the portion of resources that each application can get.

The Scheduler further organizes applications into "queues" and equitably shares resources among those queues. The default. All users share a separate queue called "Default" .

Suppose an application specifically lists a queue in a container resource request, and the request is submitted to that queue. It is also possible to allocate a queue based on the username word configured by the request.

In each queue, a scheduling policy is used to share resources between executing applications. The default is a fair memory-based dispatch. However, multiple resources for FIFO and dominant Resource fairness will also be configured. Queues are organized in a hierarchy, sharing cluster allocation resources and weight configurations at a specified scale.

In addition to providing fair sharing. Fair Scheduler agrees to assign a minimum share guarantee to the queue, which is useful for ensuring that the identified users, groups, or production applications always get sufficient resources. When a queue includes an application, it can at least get the minimum share, but when the queue does not need to share all of its guarantees. The excess will be allocated in other execution applications.

When these queues do not include an app, This allows the scheduler to ensure that the queue capacity is used efficiently at the same time.

Fair Scheduler allows all applications to execute by default, but it is also possible to limit the number of applications per user and per queue execution through the configuration file. When a user is required to submit hundreds of apps immediately. Or assuming that too many apps are performing at the same time to improve performance, which can result in too much intermediate data or too much context switching. It's very practical. Applying a restriction does not cause any subsequent submitted application to fail. Only waits in the scheduler queue until some of the users ' previous apps have finished executing.

Pluggable Hierarchy Queue

The Fair scheduler supports hierarchy queues. All queues are inherited from a queue called "root". In a typical dispatch style. The available resources are distributed in the child queues of the root queue. And then. The child queues distribute resources in the same way to their child queues. The app may only be scheduled in the left queue. Queues are designated as sub-queues for other queues, and are placed in a child element of their parent queue by placing them in the fair Dispatcher allocation file.

The name of a queue starts with the name of its parent queue, with a period (?) . as a The separator character. So a queue called "Queue1" under the root queue. will be written as "Root.queue1", in a queue called " parent1 " called " Queue2" will be written as "Root.parent1.queue2". When to participate (?) Queue, the root section is optional, so queue1 can only be written as "queue1", Queue2 is only written as "Parent1.queue2".

and there's. The Fair Scheduler agrees to set a policy of its own definition for each queue, agreeing that the user can use the queue resource in whatever way he or she wants. A self-defined policy can be inherited by org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy 来创建。 fifopolicy, Fairsharepolicy (default), and Dominantresourcefairnesspolicy is built-in, Ready for use.

in the original (MR1) Fair Scheduler in the Some attachments have not yet been supported. In some applications, you define policy management priorities as "push".

(?

)

installation

to use Fair Scheduler. First , assign the appropriate scheduler class in Yarn-site.xml :

<property>  <name>yarn.resourcemanager.scheduler.class</name>  <value> Org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.fairscheduler</value></property>

Configuration

your own definition of Fair Scheduler generally involves changes to two files. First of all. The scheduler-side option can be set by adding configuration properties to the Yarn-site.xml in your existing configuration folder. Second, most of the cases. Users will want to create a list of allocation files that include the weights and capacities of each queue . The allocation file is overloaded every 10 seconds, agreeing to dynamic changes.

properties can be placed in the Yarn-site.xml

• Yarn.scheduler.fair.allocation.file
  • Assignment (config) The path to the file. An allocation file is an XML listing that describes the queue and its attributes, as well as the default policy. This file must be the XML format described in the next section. Given a relative path, the file is searched on classpath (typically including the Hadoop conf folder).

    Default is fair-scheduler.xml.

• Yarn.scheduler.fair.user-as-default-queue
  • The name of the queue is not specified, regardless of whether the username associated with the assignment is used as the default queue name.

    If this is set to false or not set. All jobs share a default queue called "Default". The default is true.

• Yarn.scheduler.fair.preemption

  • Note that preemption is experimental in the current version number. The default is False.

• Yarn.scheduler.fair.sizebasedweight
  • Whether the app-based scale allocation is shared to individual apps. or share the same size for all applications, regardless of their size. If set to True. Assigns a weight to the app by dividing the natural logarithm of the total memory requested by one application. The default is 2.
• Yarn.scheduler.fair.assignmultiple
  • Whether to agree to a heartbeat in multiple containers allocated. The default is False.
• Yarn.scheduler.fair.max.assign
  • Assume that multiple allocations are true. The maximum number of containers will be assigned a heartbeat. The default is-1, which means no limit.
• Yarn.scheduler.fair.locality.threshold.node
• yarn.scheduler.fair.locality.threshold.rack
  • For applications that request containers on a particular rack. The number of scheduling opportunities that are allocated from the previous container to the wait and the other node.

    The floating-point number, recorded as 0-1, is the scale of the cluster. Is the number of scheduling opportunities that are passed up. The default value is-1, which means not passing whatever scheduling opportunity.

format of the allocation file

The allocation file must be in XML format. This format consists of five elements:

• Queue Element , which represents the queue. Each of these may include the following attributes:
  • minresources:minimum resources the queue is entitled to, in the form "X MB, Y vcores". For the Single-resource fairness Policy, the Vcores value is ignored. If a queue ' s minimum share is not satisfied, it'll be offered available resources before any other queue under the same Parent. Under the Single-resource fairness Policy, a queue is considered unsatisfied if it memory usage is below it minimum memo Ry share. Under dominant resource fairness, a queue is considered unsatisfied if it usage for it dominant resource with respect to The cluster capacity is below it minimum share for that resource. If multiple queues is unsatisfied in this situation, resources go to the queue with the smallest ratio between relevant R Esource usage and minimum. Note that it's possible that a queue, that's below it minimum may isn't immediately get up to its minimum when it submits An application, because already-running jobs could be using those resources.
  • Maxresources:maximum Resources A queue is allowed, in the form "X MB, Y vcores". For the Single-resource fairness Policy, the Vcores value is ignored. A queue would never be assigned a container the would put its aggregate usage over this limit.
  • Maxrunningapps:limit the number of apps from the "queue to run" at once
  • Weight:to share the cluster non-proportionally with other queues. Weights default to 1, and a queue with weight 2 should receive approximately twice as many resources as a queue with the D Efault weight.
  • schedulingPolicy : To set the scheduling policy of any queue. The Allowed values is "FIFO"/"fair"/"DRF" or any class that Extendsorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.schedulingpolicy . Defaults to "fair". If "FIFO", apps with earlier submit times is given preference for containers, but apps submitted later may run concurrent ly if there is leftover space on the cluster after satisfying the earlier app ' s requests.
  • ACLSUBMITAPPS:A List of users the can submit apps to the queue. A (default) value of "*" means that any users can submit apps. A queue inherits the ACL of its parent, so if a queue2 descends from queue1, and User1 are in queue1 ' s ACL, and User2 are in Queue2 ' s ACL, then both users could submit to queue2.
  • Minsharepreemptiontimeout:number of seconds the queue is under it minimum share before it'll try to preempt containers To take the resources from the other queues.
• The user element , which represents the settings that govern the behavior of individual users. They can include a separate attribute: Maxrunningapps, which specifies the number of user execution limits.

• a usermaxappsdefault element , assuming that there is no special designation, sets the default execution limit for each user.
• a fairsharepreemptiontimeout element , one queue before attempting to preempt a resource container from another queue. The number of seconds it is fair to share.
• a defaultqueueschedulingpolicy element, the default scheduling policy is set for the queue: it is assumed that the specified in the Schedulingpolicy element of each queue is overwritten.

  The default is "fair".

  The following is a sample of the allocation file:

  < Span style= "Background-color:inherit" ><? 
   
   
   XML version= "1.0"?><allocations> <queue name= "Sample_queue" > <minresources>10000 mb,0vcores </minResources> <maxresources>90000 mb,0vcores</maxresources> <maxrunningapps>50</ maxrunningapps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> <que UE name= "Sample_sub_queue" > <minresources>5000 mb,0vcores</minresources> </queue> </queue > <user name= "sample_user" > <maxRunningApps>30</maxRunningApps> </user> <usermaxappsd Efault>5</usermaxappsdefault></allocations> 
     

  Attention. Because of the previous Fairscheduler compatibility, the "queue" element can be replaced by the "pool" element.

Management

The Fair Scheduler provides support for two mechanisms for execution-time management:

• By editing the allocation file, you can change the minimum share, limit, weight, pre-occupancy time difference, and queue scheduling policy.

  The scheduler will reload the file 10-15 seconds after it knows it has changed.

• The current app, queue, and fair share can be checked through the ResourceManager Web interface, which is http://ResourceManager URL/cluster/scheduler.

  Each of the following queue domains can be viewed on the Web interface:

• Used resources-the sum of the resources allocated to the container within the queue.
• Number of live apps-the number of apps in the queue that received at least one container.
• Number of apps to be executed-the number of apps in the queue that have not yet been received regardless of the container.

• Minimum resource-The minimum resource that is guaranteed to be configured for a queue.
• Maximum resource-the maximum resource that has been configured to give a queue.
• Fair share-the fair sharing of resources in the queue. When other queues are not in use. Queues may be allocated more than their fair share of resources.

  The resource consumption of a queue is equal to or less than its fair share, and its container will never pre-occupy (?).

  ResourceManager to correctly display more information for each application. The web includes an application interface for fair sharing.

Hadoop-2.2.0 China document--mapreduce Next Generation--fair dispatch