Spark core source code analysis: spark task running model

Dagscheduler

The stage-oriented scheduling layer generates a DAG composed of stages for the job and submits the taskset to taskscheduler for running.

Each stage is an independent task. They run the same compute function and enjoy the same shuffledependencies. When splitting the stage, the Dag follows the shuffle line.

private[spark]class DAGScheduler(    taskScheduler: TaskScheduler,    listenerBus: LiveListenerBus,    mapOutputTracker: MapOutputTrackerMaster,    blockManagerMaster: BlockManagerMaster,    env: SparkEnv)  extends Logging {

// In actor mode, receive the dagschedulerevent and perform processeventprivate var eventprocessactor: actorref = _ private [scheduler] Val nextjobid = new atomicinteger (0) Private [schedjobs] def numtotaljobs: int = nextjobid. get () Private Val nextstageid = new atomicinteger (0) // a series of information maintenance, very clear Private [schedinteger] Val jobidtostageids = new hashmap [int, hashset [int] private [schedstage] Val stageidtojobids = new hashmap [int, hashset [int] private [schedmap] Val stageidtostage = new hashmap [int, stage] private [scheduler] Val shuffletomapstage = new hashmap [int, stage] private [scheduler] Val jobidtoactivejob = new hashmap [int, activejob] private [scheduler] Val resultstagetojob = new hashmap [stage, activejob] private [scheduler] Val stagetoinfos = new hashmap [stage, stageinfo] // maintain stages in different States, very clear // stages we need to run whose parents aren't done Private [schedages] Val waitingstages = new hashset [stage] // stages we are running right now Private [scheduler] Val runningstages = new hashset [stage] // stages that must be resubmitted due to fetch failures private [schedures] Val failedstages = new hashset [stage] // missing tasks from each stage private [scheduler uler] Val pendingtasks = new hashmap [stage, hashset [task [_] private [scheduler] Val activejobs = new hashset [activejob] // contains the locations that each RDD's partitions are cached on private Val cachelocs = new hashmap [int, array [seq [tasklocation]

The actor is initialized in the START () method and then receives the dagschedulerevent processing. Schedcontext starts in sparkcontext.

Event Processing

Source code reading entry: it can be expanded based on the processevent (Event: dagschedulerevent) method.

The processed events include the following:

Submit job

Jobsubmitted event:

The number of tasks passed in by task submission events is as follows:

case JobSubmitted(jobId, rdd, func, partitions, allowLocal, callSite, listener, properties)

The processing process can be divided into three steps. The detailed logic involved in each step is further described below.

finalStage = newStage(rdd, partitions.size, None, jobId, Some(callSite))

This newstage () operation can correspond to a new result stage or shuffle stage. Returns the stage class (which records some information ). The stage class will pass in the option [shuffledependency [_, _] Number of workers. There is an isshufflemap variable inside to identify whether the stage is shuffle or result.

val job = new ActiveJob(jobId, finalStage, func, partitions, callSite, listener, properties)

The activejob class also records some information and can be considered as a Vo class.

if (allowLocal && finalStage.parents.size == 0 && partitions.length == 1) {// Compute very short actions like first() or take() // with no parent stages locally.listenerBus.post(SparkListenerJobStart(job.jobId, Array[Int](), properties))runLocally(job)} else {jobIdToActiveJob(jobId) = jobactiveJobs += jobresultStageToJob(finalStage) = joblistenerBus.post(SparkListenerJobStart(job.jobId, jobIdToStageIds(jobId).toArray, properties))submitStage(finalStage)}

First, it is inferred that the stage has no parent dependency, and if the partition is 1, the local task is run. Otherwise, submitstage.

 

The logic of submitstage is: first look for the parents of this stage. If there is no missing parent stage, submitmissingtask is used to submit the task of this stage. Assume that the parent stage is recursively submitstage, and the result set obtained by getmissingparentstages is sorted in descending order by ID, that is, the recursive submitstage is performed in the order of the parent stage IDs.

 

Submitmissingtask processes the stage in which the parent of the stage has been available. The main logic is as follows:

Step 1: use stage. isshufflemap to determine whether to generate shufflemaptask or resulttask. The number of shufflemaptasks generated is equal to the number of partition.

Step 2: Build the generated tasks into a taskset and submit it to the submittasks method of taskscheduler.

Taskscheduler

Dagscheduer, in the unit of stage, provides tasks to taskscheduer and the implementation class is taskschedulerimpl.

 

Taskschedulerimpl:

Schedulerbackend

Schedulablebuilder

Dagscheduler

Tasksetmanager

Taskresultgetter

Tasks information (taskidtotasksetid, taskidtoexecutorid, activeexecutorids)

Other information (schedulermode)

 

Taskschedtor receives tasks, receives resources and executors, maintains information, deals with backend, and distributes tasks.

 

Start (), stop (), start (), stop () of backend ()

Submittasks (taskset) logic:

Generate a new tasksetmanager for these tasks, add tasksetmanager to schedulerbuilder, and then perform a reviveoffer () operation to the backend.

Schedulerbuilder

Schedulablebuilder has two implementations: FIFO and fair. addtasksetmanager adds tasksetmanager to the pool. In FIFO mode, there is only one pool. Fair has multiple pools, and the pool is also divided into FIFO and fair modes.

The pool or tasksetmanager can be added to the rootpool of schedulablebuilder. Both are inherited classes of scheduable. Therefore, schedulablebuilder is used to maintain the scheduable tree structure of the rootpool. The pool is a non-leaf node on the tree, and tasksetmanager is a leaf node.

Builddafaultpool is used during taskscheduler initialization.

Tasksetmanager

Tasksetmanager is responsible for the startup, retry upon failure, and localization of these tasks. Each time the reourseoffer method finds a suitable task (execid, host, locality) and starts it.

 

Reourseoffer method,

  def resourceOffer(      execId: String,      host: String,      maxLocality: TaskLocality.TaskLocality)

Find the task that matches execid, host, and locality. If you find the task, start the task. At startup, add the task to the hashset of runningtask and call the taskstarted method of dagscheduler. The taskstarted method sends the dagschedulerevent of beginevent to eventprocessoractor.

Taskresultgetter

Maintain a thread pool for deserialization and obtaining task results from the remote end.

def enqueueSuccessfulTask(taskSetManager: TaskSetManager, tid: Long, serializedData: ByteBuffer)

After deserialization of serialized data is parsed, there are two situations: directly readable result and indirect task result.

The former is the directtaskresult [T] class:

class DirectTaskResult[T](var valueBytes: ByteBuffer, var accumUpdates: Map[Long, Any], var metrics: TaskMetrics)

The latter is the indirecttaskresult [T] class:

case class IndirectTaskResult[T](blockId: BlockId) extends TaskResult[T] with Serializable

After the indirecttaskresult is parsed, you can obtain the blockid class, which has the following implementations:

In taskresultgetter, The getremotebytes (blockid) method of blockmanager is used to obtain the serialized task result. After the task result is parsed, The directtaskresult class is obtained, to obtain the real result data after deserialization.

This is a rough process. For example, different events will be sent to scheduler, and blockmanager will call blockmanagermaster to remove the block.

 

The blockid class has the following key variables:

private[spark] sealed abstract class BlockId {  /** A globally unique identifier for this Block. Can be used for ser/de. */  def name: String  // convenience methods  def asRDDId = if (isRDD) Some(asInstanceOf[RDDBlockId]) else None  def isRDD = isInstanceOf[RDDBlockId]  def isShuffle = isInstanceOf[ShuffleBlockId]  def isBroadcast = isInstanceOf[BroadcastBlockId]

The following describes how blockmanager obtains data through blockid:

The internal method of blockmanager is called.

Private def dogetremote (blockid: blockid, asvalues: Boolean): Option [any] = {require (blockid! = NULL, "blockid is null") // obtain the locations Val locations = random of this blockid through blockmanagermaster. shuffle (master. getlocations (blockid) for (loc <-locations) {logdebug ("getting remote block" + blockid + "from" + LOC) // use blockmanagerworker to obtain the block data VAL data = blockmanagerworker. syncgetblock (getblock (blockid), connectionmanagerid (loc. host, loc. port) if (Data! = NULL) {If (asvalues) {// return some (datadeserialize (blockid, data)} else {return some (data )}} logdebug ("the value of block" + blockid + "is null")} logdebug ("Block" + blockid + "not found") None}

The idea is to obtain the block location information through blockmanagermaster. After the collection is disrupted, the location information is traversed and the data is obtained through blockmanagerworker. If only the data is obtained, the data is returned after deserialization.

 

During taskresultgetter processing, the handlesuccessfultask and handlefailedtask methods are called to Scheduler for success and failure respectively.

Handlesuccessfultask will issue a completionevent event in dagscheduler. There will be a lot of details at the end of this step. I will not read it here.

Handlefailedtask only requires tasksetmanager instead of zombie. if the task is not killed, it will continue to call backend. reviveoffers () to re-run.

Full Text :)

Spark core source code analysis: spark task running model