Spark scheduling policy details

Spark scheduling policy details
Spark scheduling policy

Spark currently has two Scheduling Policies: FIFO (first come first served) and FAIR (FAIR strategy. The so-called scheduling policy is to sort the scheduled objects and schedule them by priority. The scheduling sorting interface is as follows, which compares two schedulable objects.

private[spark] trait SchedulingAlgorithm { def comparator(s1: Schedulable, s2: Schedulable): Boolean} 

Its implementation classes are kerberoschedulingalgorithm and FairSchedulingAlgorithm.

/*** Implement FIFO sorting. The main factor is the priority, followed by the priority of the corresponding Stage *. The priority is the same, the front stage takes precedence */private [spark] class into oschedulingalgorithm extends SchedulingAlgorithm {override def comparator (s1: Schedulable, s2: Schedulable ): boolean = {// generally, the smaller the priority, the higher the priority. val priority1 = s1.priority val priority2 = s2.priority var res = math. signum (priority1-priority2) if (res = 0) {// if the priority is the same, the first priority of the Stage is val stageId1 = s1.stageId val stageId2 = s2.stageId res = math. signum (stageId1-stageId2)} if (res <0) {true} else {false }}}

Note:
You can override this comparison method based on your own definition of the priority. However, if the priority and Stage are the same, the comparison method is later by default.

Private [spark] class FairSchedulingAlgorithm extends SchedulingAlgorithm {override def comparator (s1: Schedulable, s2: Schedulable): Boolean ={// minimum share, it can be understood that the minimum resource required for execution is the number of CPU cores, val minShare1 = s1.minShare val minShare2 = s2.minShare // Number of running tasks val runningTasks1 = interval val runningTasks2 = s2.runningTasks // whether there are ELE. Me tasks, check whether the number of allocable cores is less than the number of tasks. if the resources are insufficient, the system will be hungry. val s1Needy = runningT Asks1 <minShare1 val s2Needy = runningTasks2 <minShare2 // minimum resource usage ratio. Here, we can consider it as val minShareRatio1 = runningTasks1.toDouble/math, Which is lighter. max( minShare1, 1.0 ). toDouble val minShareRatio2 = runningTasks2.toDouble/math. max (minShare2, 1.0 ). toDouble // weight, the number of tasks is the same, the priority of the high weight is val taskToWeightRatio1 = runningTasks1.toDouble/s1.weight. toDouble val taskToWeightRatio2 = runningTasks2.toDouble/s2.weight. toDouble Var compare: Int = 0 // hunger first if (s1Needy &&! S2Needy) {return true} else if (! S1Needy & s2Needy) {return false} else if (s1Needy & s2Needy) {// if both are hungry, compare = minShareRatio1.compareTo (minShareRatio2) is preferred )} else {// do not starve, then compare the weight ratio, with a lower proportion priority compare = taskToWeightRatio1.compareTo (taskToWeightRatio2)} if (compare <0) {true} else if (compare> 0) {false} else {// if the names are the same, the names are compared in alphabetical order without considering the length. Therefore, the names are more important. s1.name <s2.name }}}

Note:

1. The principle of fairness is based on the principle that whoever needs it most, so the people who suffer from hunger are given priority;

2. Resource occupation is a bit confusing. If you think of him as a greedy problem, it is easy to understand. This can be understood for tasks that are all out of hunger. You may not be able to effectively relieve resources that are provided by high loads, and you may not be able to quickly use them if the load is small, more resources can be released after completion, which is a greedy strategy. For example, the number of jobs in JobA and JobB is the same as 10, the minShare of A is 2, and the share of B is 5. The share ratio is 5 and 2. Obviously, the share of B is smaller, greedy policies should be scheduled for B;

3. For those who are in the satisfied state, of course, their weights are better decisive, and their weights are higher than those who are low (larger than their rights );

4. If all the above comparisons are the same, the top priority of name dictionary sorting (haha, the name is very important); The name aaa takes precedence over abc, therefore, we need to consider this when naming the Pool or TaskSetManager.

The two scheduling sorting algorithms target comparable objects which are specific to Schedule. The definition of trait (which can be understood as an interface in java) is as follows:

Private [spark] trait Schedulable {// specifies the parent object, that is, the scheduling object to which the Pool or TaskSetManager belongs. Scheduling is hierarchical and tree-like var parent: pool // Its scheduling object, that is, the scheduling object that is responsible for management def schedulableQueue: concurrent1_queue [Schedulable] // The sorting model between managed objects, currently, only the FIFO and FAIR algorithms def schedulingMode: SchedulingMode // weight refers to the weight of the same level. The larger the weight, the more resources are obtained. def weight: int // minimum share value refers to the minimum number of resources required for running, that is, number of CPUs def minShare: Int def runningTasks: Int // priority, it refers to the priority in the same level. The priority scheduling def priority: Int // This stageId is for TaskSetManager. Because of the Tasks of a Stage, def stageId is actually submitted using a TaskSet: int def name: String def addSchedulable (schedulable: Schedulable): Unit def evaluate (name: String): Schedulable def executorLost (executorId: String, host: String): Unit def checkSpeculatableTasks (): Boolean def getSortedTaskSetQueue: ArrayBuffer [TaskSetManager]}

Currently, Spark has two types of schedulable entities: Pool and TaskSetManager. A Pool is a scheduling Pool with sub-pools. The rootPool in Spark is an unknown Pool by default for the root node.

Please click: http://click.aliyun.com/m/8868/