Spark: source code analysis submitted by tasks to executor

From org. Apache. Spark. schedks. dagschedks # submitmissingtasks, analyze how the stage generates taskset.

If all the parent stages of a stage have been computed or exist in the cache, submitmissingtasks will be called to submit the tasks contained in the stage.

Org. Apache. Spark. schedks. dagschedks # The submitmissingtasks calculation process is as follows:

1. First, get the partition to be calculated in RDD. For a shuffle stage, you need to determine whether the result is cached in the stage. For final stage of the result type, determine whether the partition in the computing job has been completed.
2. Serialize the binary of a task. Executor can be obtained through broadcast variables. Each task is deserialized first. In this way, tasks run on different executors are isolated and will not affect each other.
3. Generates a task for each partition to be calculated: for the stage dependent on the shuffle type, generates the shufflemaptask type. For the result stage, generates a resulttask type task.
4. Make sure that tasks can be serialized. Because different clusters have different taskscheduler, the logic can be simplified here, And the taskset tasks can be serialized.
5. Submit taskset through taskscheduler.

Taskset is a set of identical tasks that can be used as pipelines. Each task has the same processing logic. The difference is that it processes data. Each task is responsible for processing a partition. Pipeline, which can be called the cornerstone of big data processing, can be put into the cluster for running only when data is pipeline processed. For a task, it obtains the logic from the data source and then executes the logic in the topological order (actually calling the RDD compute ). Taskset is a data structure that stores this group of tasks: [Java]View plaincopy

1. Private [spark] class taskset (
2. Val tasks: array [task [_],
3. Val stageid: int,
4. Val attempt: int,
5. Val priority: int,
6. Val properties: Properties ){
7. Val id: String = stageid + "." + Attempt
9. Override def tostring: String = "taskset" + id
10. }

When you manage and schedule this taskset, org. Apache. Spark. scheduler. tasksetmanager is responsible for retry of task failures, tracking the execution status of each task, and processing the call of locality-aware. The detailed call stack is as follows:

1. Org. Apache. Spark. scheduler. taskschedulerimpl # submittasks
2. Org. Apache. Spark. scheduler. schedulablebuilder # addtasksetmanager
3. Org. Apache. Spark. schedend. Cluster. coarsegrainedschedulerbackend # reviveoffers
4. Org. Apache. Spark. schedend. Cluster. coarsegrainedschedulerbackend. driveractor # makeoffers
5. Org. Apache. Spark. scheduler. taskschedulerimpl # resourceoffers
6. Org. Apache. Spark. schedend. Cluster. coarsegrainedschedulerbackend. driveractor # launchtasks
7. Org.apache.spark.exe cutor. coarsegrainedexecutorbackend. receivewithlogging # launchtask
8. Org.apache.spark.exe cutor. executor # launchtask

First click org.apache.spark.exe cutor. executor # launchtask: [Java]View plaincopy

1. Def launchtask (
2. Context: executorbackend, taskid: Long, taskname: String, serializedtask: bytebuffer ){
3. Val TR = new taskrunner (context, taskid, taskname, serializedtask)
4. Runningtasks. Put (taskid, TR)
5. Threadpool.exe cute (TR) // start running in executor
6. }

Taskrunner deserializes the task from the serialized task. For details, see org.apache.spark.exe cutor. executor. taskrunner # Run: task. Run (taskid. toint ). The implementation of task. Run is: [Java]View plaincopy

1. Final def run (attemptid: Long): t = {
2. Context = new taskcontext (stageid, partitionid, attemptid, runninglocally = false)
3. Context. taskmetrics. hostname = utils. localhostname ()
4. Taskthread = thread. currentthread ()
5. If (_ killed ){
6. Kill (interruptthread = false)
7. }
8. Runtask (context)
9. }

For the two previously mentioned tasks, that is

1. Org. Apache. Spark. schedtask. shufflemaptask
2. Org. Apache. Spark. schedask. resulttask

Different runtasks are implemented: org. Apache. Spark. scheduler. resulttask # runtasks call the compute of RDD sequentially, And the partition is calculated through the topological order of RDD:
[Java]View plaincopy

1. Override def runtask (Context: taskcontext): U = {
2. // Deserialize the RDD and the func using the broadcast variables.
3. Val SER = sparkenv. Get. closureserializer. newinstance ()
4. Val (RDD, func) = Ser. deserialize [(RDD [T], (taskcontext, iterator [T]) => U)] (
5. Bytebuffer. Wrap (taskbinary. value), thread. currentthread. getcontextclassloader)
7. Metrics = some (context. taskmetrics)
8. Try {
9. Func (context, RDD. iterator (partition, context ))
10. } Finally {
11. Context. marktaskcompleted ()
12. }
13. }

While org. Apache. Spark. schedle. shufflemaptask # runtask is the result of shuffle writing, [Java]View plaincopy

1. Override def runtask (Context: taskcontext): mapstatus = {
2. // Deserialize the RDD using the broadcast variable.
3. Val SER = sparkenv. Get. closureserializer. newinstance ()
4. Val (RDD, DEP) = Ser. deserialize [(RDD [_], shuffledependency [_, _, _])] (
5. Bytebuffer. Wrap (taskbinary. value), thread. currentthread. getcontextclassloader)
6. // Taskbinary is obtained from the broadcast variable of the task serialized in org. Apache. Spark. schedks. dagscheduler # submitmissingtasks.
8. Metrics = some (context. taskmetrics)
9. VaR Writer: shufflewriter [Any, any] = NULL
10. Try {
11. Val manager = sparkenv. Get. shufflemanager
12. Writer = manager. getwriter [Any, any] (Dep. shufflehandle, partitionid, context)
13. Writer. Write (RDD. iterator (partition, context). asinstanceof [iterator [_ <: product2 [Any, any]) // write the result of RDD calculation to memory or disk
14. Return writer. Stop (success = true). Get
15. } Catch {
16. Case E: exception =>
17. If (writer! = NULL ){
18. Writer. Stop (success = false)
19. }
20. Throw E
21. } Finally {
22. Context. marktaskcompleted ()
23. }

Neither of the two tasks calls the RDD compute in the topological order to compute the partition. The difference is that shufflemaptask needs to shuffle write for the Child stage to read the shuffle result. Taskbinary used by both tasks is obtained as the broadcast variable of the task serialized in org. Apache. Spark. scheduler. dagscheduler # submitmissingtasks.

Spark: source code analysis submitted by tasks to executor