Spark 1.60 of Executor schedule

The first time to see the source code or spark 1.02. This time see Xinyuan code Discovery Dispatch mode has some new features, here casually write about.

Unchanged, master still receives appclient and worker messages and executes schedule () after receiving messages such as RegisterApplication. Schedule () will still find the idle worker to perform the waitingdrivers first. But the way to dispatch executor has changed a little.

1   PrivateDef startexecutorsonworkers (): Unit = {2     //very simple FIFO scheduler. We keep trying to fit in the first app3     //In the queue, then the second app, etc.4      for(App <-WaitingappsifApp.coresleft > 0) {5Val Coresperexecutor:option[int] =App.desc.coresPerExecutor6       //Filter out workers that don ' t has enough resources to launch an executor7Val usableworkers = workers.toArray.filter (_.state = =workerstate.alive)8. Filter (worker = Worker.memoryfree >= App.desc.memoryPerExecutorMB &&9Worker.coresfree >= Coresperexecutor.getorelse (1))Ten . SortBy (_.coresfree). Reverse OneVal assignedcores =scheduleexecutorsonworkers (app, Usableworkers, Spreadoutapps) A  -       //Now the we ' ve decided how many cores to allocate on each worker and let's allocate them -        for(POS <-0 until Usableworkers.lengthifAssignedcores (POS) > 0) { the Allocateworkerresourcetoexecutors ( - app, Assignedcores (POS), Coresperexecutor, Usableworkers (POS)) -       } -     } +}

As we can see, although this is still the simple FIFO scheduling, but no longer is the core of the scheduling, will adapt to the core number of executor requirements, in the search for Usableworkers will find memory and cores meet the conditions of the worker. This is to change one of the previous bugs, namely:

Cluster has a 4 workers with a cores each. User Requests 3 executors (Spark.cores.max = Spark.executor.cores = 16). If 1 Core is allocated at a time, the cores from each worker would being assigned to each executor. Since <, no executors would launch [SPARK-8881]

Where Scheduleexecutorsonworkers is the number of cores that are found for each available worker assigned to the current app.

In short, this is still a simple FIFO scheduling, compared to yarn default capacity a lot less features, and Google's Borg paper "Large-scale cluster management at Google with Borg" The details of the dispatch are more complex, but the win is easy to understand, look at the code to ease a lot ...

Spark 1.60 of Executor schedule