Spark's RDD checkpoint implementation analysis

Overview

In the "in-depth understanding of spark: core ideas and source analysis," a simple introduction of the next Rdd checkpoint, the book is a pity. So the purpose of this article is to check the gaps and improve the contents of this book.

Spark's Rdd will save checkpoints after execution, so that when the entire job fails to run again, the successful RDD results have been run before recovery from the checkpoint, which will significantly reduce the cost of recalculation, improve task recovery efficiency and execution efficiency. Save resources for each compute node in spark. This article focuses on the analysis of the code implementation of checkpoints, more in-depth understanding of its principles. in the 5th chapter of "In-depth understanding of spark: core ideas and source analysis", the partitions method is called when obtaining the RDD partition information (see listing 5-11), and the dependencies method is called when obtaining a dependency (see Code Listing 5-28).

Code Listing 5-11 implementation of the Partitions method

  Final def partitions:array[partition] = {    Checkpointrdd.map (_.partitions). Getorelse {      if (partitions_ = = null) {        partitions_ = getpartitions      }      partitions_    }  }

Code Listing 5-28 implementation of the Dependencies method

  Final Def dependencies:seq[dependency[_]] = {    Checkpointrdd.map (r = = List (new onetoonedependency (R))). Getorelse {      if (Dependencies_ = = null) {        Dependencies_ = getdependencies      }      dependencies_    }  }

The partitions method of the RDD shown in listing 5-11 and the dependencies method of the rdd shown in Listing 5-28 are used in the CHECKPOINTRDD,CHECKPOINTRDD definition as follows.

  Private Def checkpointrdd:option[rdd[t]] = Checkpointdata.flatmap (_.checkpointrdd)

From the definition of Checkpointrdd, Checkpointrdd's information comes from the definition of Checkpointdata,checkpointdata as follows:

  Private[spark] var checkpointdata:option[rddcheckpointdata[t]] = None

So Checkpointrdd is actually obtained by invoking the Checkpointrdd method of Rddcheckpointdata, the implementation of CHECKPOINTRDD is shown in Listing 5-63.

Code Listing 5-63 getting checkpoint data

  def checkpointrdd:option[rdd[t]] = {    rddcheckpointdata.synchronized {      cprdd    }  }

Cprdd holds the Checkpointrdd after the checkpoint is saved, and its type is defined as follows:

  var cpstate = Initialized  @transient var cpfile:option[string] = none  var cprdd:option[rdd[t]] = None

In addition to showing the type of Cprdd, the above code defines cpfile and Cpstate. Cpfile is used to save a checkpoint to the file directory that is written to HDFs, Cpstate is used to represent the state of the current RDD set checkpoint, including initialization (Initialized), Mark to save checkpoints (Markedforcheckpoint), Save Checkpoints (checkpointinginprogress) and set checkpoint completion (checkpointed) in several states. The Cpstate field of Rddcheckpointdata is the initialized state by default. Check Point is enabled

A user-submitted spark job must proactively invoke the checkpoint method of the RDD (see listing 5-64) to start the checkpoint feature.

Code listing 5-64 enabling the checkpoint feature

  Def checkpoint () {    if (context.checkpointDir.isEmpty) {      throw new Sparkexception ("Checkpoint directory has not been set in the Sparkcontext ")    } else if (checkpointdata.isempty) {      checkpointdata = Some (new Rddcheckpointdata (t His))      CheckpointData.get.markForCheckpoint ()    }  }

Only the checkpoint method is called, and the RDD creates the Rddcheckpointdata object and is held by Checkpointdata. The Markforcheckpoint method of Rddcheckpointdata is also called (see Listing 5-65) to set the Cpstate state to Markedforcheckpoint.

Code Listing 5-65 tags enable checkpoints

  Def markforcheckpoint () {    rddcheckpointdata.synchronized {      if (cpstate = = Initialized) Cpstate = Markedforcheckpoint    }  }

It is important to place Cpstate as Markedforcheckpoint in the Markforcheckpoint method: On the one hand, the checkpoint is enabled, and the other is only when cpstate equals Markedforcheckpoint. Before the checkpoint can be saved.

Checkpoint SaveCprdd is used to hold Checkpointrdd, but when does it hold? Below we will analyze the timing of checkpoints. In the introduction of code listing 5-21, the simple mention of invoking the Rdd Docheckpoint method to save the checkpoint, now to specifically analyze its code implementation, see listing 5-66.
Code Listing 5-66 Saving checkpoints

  Def docheckpoint () {rddcheckpointdata.synchronized {if (cpstate = = markedforcheckpoint) {cpstate = Che  Ckpointinginprogress} else {return}}//Create the output path for the checkpoint Val path =    New Path (Rdd.context.checkpointDir.get, "rdd-" + rdd.id) val fs = Path.getfilesystem (rdd.context.hadoopConfiguration) if (!fs.mkdirs (path)) {throw new Sparkexception ("Failed to create checkpoint path" + path)}//Save to fi Le, and reload it as an RDD val broadcastedconf = rdd.context.broadcast (New serializablewritable (Rdd.context.hado  opconfiguration) Rdd.context.runJob (Rdd, Checkpointrdd.writetofile[t] (path.tostring, broadcastedconf) _) Val Newrdd  = new Checkpointrdd[t] (Rdd.context, path.tostring) if (newRDD.partitions.size! = rdd.partitions.size) {throw new Sparkexception ("Checkpoint RDD" + Newrdd + "(" + NewRDD.partitions.size + ") has different" + "number of partitions than OrigiNAL Rdd "+ Rdd +" ("+ Rdd.partitions.size +")}//Change the dependencies and partitions of the RDD RDDCHEC   kpointdata.synchronized {cpfile = Some (path.tostring) Cprdd = Some (Newrdd) rdd.markcheckpointed (Newrdd) Update The Rdd ' s dependencies and partitions cpstate = checkpointed} loginfo ("Done checkpointing Rdd" + Rd D.id + "to" + Path + ", new parent is RDD" + newrdd.id)}

By reading the Docheckpoint method, its execution steps are summarized as follows:

1) Verify that the user job has checkpoints enabled, that is, if the checkpoint method is called, Cpstate is set to Markedforcheckpoint. If checkpoints are not enabled, they are returned directly without continuing the checkpoint save.

2) Create a file path on HDFs to hold the checkpoint data. Where Checkpointdir must be set by the user job calling Sparkcontext's Setcheckpointdir method (see listing 5-67).

3) Run the job, this job actually executes the Checkpointrdd WriteToFile method (see listing 5-68), and saves the checkpoint data on HDFs.

4) The constructed Checkpointrdd is held by Cprdd, the checkpoint Save directory is held by Cpfile, and finally the cpstate is set to checkpointed. Because the checkpoint is saved, the RDD has been successfully executed, its dependency and partition-related information will no longer be used, even if the job recovery only needs to read from the checkpoint, the Markcheckpointed method called the RDD (see Code Listing 5-69) clears the dependency and partition information.

Code Listing 5-67 Setting the save path for job checkpoint on HDFs

  def setcheckpointdir (directory:string) {    Checkpointdir = Option (directory). map {Dir      = val = Path = new Path (di R, Uuid.randomuuid (). toString)      val fs = Path.getfilesystem (hadoopconfiguration)      fs.mkdirs (path)      Fs.getfilestatus (path). Getpath.tostring    }  }

Code Listing 5-68 writing checkpoints to HDFs

  def Writetofile[t:classtag] (path:string, broadcastedconf:broadcast[serializablewritable[configuration]], Blocksize:int =-1) (Ctx:taskcontext, Iterator:iterator[t]) {val env = sparkenv.get val outputDir = new P Ath (path) val fs = Outputdir.getfilesystem (broadcastedConf.value.value) Val finaloutputname = Splitidtofile (ctx.part Itionid) Val Finaloutputpath = new Path (OutputDir, finaloutputname) val tempoutputpath = new Path (OutputDir, "." + F Inaloutputname + "-attempt-" + Ctx.attemptid) if (fs.exists (Tempoutputpath)) {throw new IOException ("Checkpoint F Ailed:temporary path "+ Tempoutputpath +" already exists ")} val buffersize = Env.conf.getInt (" Spark.buffe R.size ", 65536) Val fileoutputstream = if (BlockSize < 0) {Fs.create (Tempoutputpath, False, buffersize)} E LSE {//This was mainly for testing purpose Fs.create (Tempoutputpath, False, BufferSize, Fs.getdefaultreplicatio N, blockSize)} VAL serializer = env.serializer.newInstance () val serializestream = Serializer.serializestream (fileoutputstream) seria Lizestream.writeall (iterator) Serializestream.close () if (!fs.rename (Tempoutputpath, Finaloutputpath)) {if (!f S.exists (Finaloutputpath)) {loginfo ("Deleting tempoutputpath" + tempoutputpath) fs.delete (Tempoutputpath, false) throw new IOException ("Checkpoint failed:failed to save output of task:" + Ctx.attemptid + "and Final output path does not exist ")} else {//Some other copy of this task must ' ve finished before us and Ren Amed it loginfo ("Final Output path" + Finaloutputpath + "already exists; Not overwriting it ") Fs.delete (Tempoutputpath, False)}}}

Code Listing 5-69 clearing the Rdd dependencies and partitioning

  Private[spark] def markcheckpointed (Checkpointrdd:rdd[_]) {    cleardependencies ()    partitions_ = null    deps = NULL    //Forget the constructor argument for dependencies too  }  protected Def cleardependencies () {    Depe Ndencies_ = null  }

Using checkpoints

The first two sections explain how checkpoints are enabled and how they are enabled and how they are implemented, and the two usage scenarios for checkpoints are described at the beginning of section 5.7:

1) When obtaining the dependency of the RDD, if there is a checkpoint, it is read from the checkpoint;

2) When obtaining a partition of the RDD, if there is a checkpoint, it is read from the checkpoint.

In addition to the above two scenarios, there is another scenario that indirectly uses the RDD checkpoint data, which is called the Computeorreadcheckpoint method of the RDD during the calculation (see listing 5-70) to read the saved calculation directly from the checkpoint. The specific use of this method is presented in the 6th chapter of the Analysis code listing 6-1, where only the implementation of the checkpoint is analyzed.

Listing 5-70 reading the calculation results from the checkpoint

  Private[spark] def computeorreadcheckpoint (Split:partition, Context:taskcontext): iterator[t] =  {    if ( ischeckpointed) Firstparent[t].iterator (split, context) else compute (split, context)  }

The ischeckpointed here is actually a method, the code is as follows.

  def Ischeckpointed:boolean = checkpointdata.exists (_.ischeckpointed)

Ischeckpointed actually proxies the ischeckpointed method of Rddcheckpointdata (see listing 5-71) to determine whether the current RDD has a checkpoint set.

Code Listing 5-71 Determining if the RDD has maintained a checkpoint

  def Ischeckpointed:boolean = {    rddcheckpointdata.synchronized {cpstate = = checkpointed}}  

Based on the previous analysis, we know that if the checkpoint has been saved, then cpstate must be equal to checkpointed, so the Ischeckpointed method returns True. The code listing 5-70 will therefore continue to execute firstparent[t].iterator (split, context). The firstparent (see Code Listing 5-13) first invokes the dependencies method shown in listing 5-28, so that Computeorreadcheckpoint is called during the calculation, The process of using checkpoints actually degrades the way we say to use checkpoints when getting rdd dependencies. The dependency has been replaced by Checkpointrdd, iterative calculations (refer to chapter 6th), and eventually the compute method of the Checkpointrdd is called (see Listing 5-72). From its implementation it is known that reading the results from the checkpoint is actually reading the data written to HDFs before it is read.

Code Listing 5-72 reading data from an HDFs saved checkpoint

  Override Def compute (split:partition, Context:taskcontext): iterator[t] = {    val file = new Path (Checkpointpath, CHEC Kpointrdd.splitidtofile (Split.index))    checkpointrdd.readfromfile (file, broadcastedconf, context)  }

Spark's RDD checkpoint implementation analysis