Spark kernel Secret -06-tasksceduler start source parsing first experience

The TaskScheduler instance object launches the source code as follows:

As can be seen from the above code, TaskScheduler's start-up is in Sparkcontext

Find the Start method implementation in the Taskschedulerimpl implementation class:

1, from the above code to see, first start Coarsegrainedschedulerbackend,

From the above Coarsegrainedschedulerbackend class code, you can see that spark started the Driveractor, named Coarsegrainedscheduler, which is a Akka message communication class, The Prestart () method is run first

From the above code can be seen, the initialization of the Akka client monitoring, and most importantly called the system's scheduler scheduling, the parameter function is to execute the schedule immediately, interval 1000 milliseconds, run Reviveoffers method

Enter the Makeoffers () method:

To run the Launchtask method:

This code is the Spark serial number task size exceeds akkaframesize-akkautils.reservedsizebytes size, then the error is "serialized task%s:%d was%d bytes, which Exceeds Max allowed: "+
"Spark.akka.frameSize (%d bytes)-Reserved (%d bytes). Consider increasing "+
"Spark.akka.frameSize or using broadcast variables for large values." , the task is now terminated and the task is removed from the task list, so it is recommended to use broadcast broadcast mode

Otherwise, it will get the execution of the task data and reduce the number of idle CPUs, send message execution Launchtask (new Serializablebuffer (Serializedtask)) method, That is, the Launchtask method of the Coarsedgrainedexecutorbackend class:

The above code will run the executor Launchtask method:

Taskrunner is a multi-threaded:

Too much code, I can not, in fact, according to the machine condition, running task tasks

2. then we go back to the Start method in the Taskschedulerimpl implementation class

If Islocal=false and spark.speculation=true, not local mode, it is necessary to dispatcher distribution tasks, the default is 100 milliseconds immediately after the start, and the interval of 100 milliseconds loop run,

Coarsegrainedschedulerbackend's reviveoffers:

Spark kernel Secret -06-tasksceduler start source parsing first experience