Spark kernel Secret -06-tasksceduler start source parsing first experience

Source: Internet
Author: User

The TaskScheduler instance object launches the source code as follows:

As can be seen from the above code, TaskScheduler's start-up is in Sparkcontext

Find the Start method implementation in the Taskschedulerimpl implementation class:

1, from the above code to see, first start Coarsegrainedschedulerbackend,

From the above Coarsegrainedschedulerbackend class code, you can see that spark started the Driveractor, named Coarsegrainedscheduler, which is a Akka message communication class, The Prestart () method is run first

From the above code can be seen, the initialization of the Akka client monitoring, and most importantly called the system's scheduler scheduling, the parameter function is to execute the schedule immediately, interval 1000 milliseconds, run Reviveoffers method

Enter the Makeoffers () method:

To run the Launchtask method:

This code is the Spark serial number task size exceeds akkaframesize-akkautils.reservedsizebytes size, then the error is "serialized task%s:%d was%d bytes, which Exceeds Max allowed: "+
"Spark.akka.frameSize (%d bytes)-Reserved (%d bytes). Consider increasing "+
"Spark.akka.frameSize or using broadcast variables for large values." , the task is now terminated and the task is removed from the task list, so it is recommended to use broadcast broadcast mode

Otherwise, it will get the execution of the task data and reduce the number of idle CPUs, send message execution Launchtask (new Serializablebuffer (Serializedtask)) method, That is, the Launchtask method of the Coarsedgrainedexecutorbackend class:

The above code will run the executor Launchtask method:

Taskrunner is a multi-threaded:

Too much code, I can not, in fact, according to the machine condition, running task tasks

2. then we go back to the Start method in the Taskschedulerimpl implementation class

If Islocal=false and spark.speculation=true, not local mode, it is necessary to dispatcher distribution tasks, the default is 100 milliseconds immediately after the start, and the interval of 100 milliseconds loop run,

Coarsegrainedschedulerbackend's reviveoffers:

Spark kernel Secret -06-tasksceduler start source parsing first experience

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.