Sparkcontext.setcheckpointdir ()

class sparkcontext extends Logging with executorallocationclient

Main entry point for Spark functionality.

The main entry for the Spark function function.

def parallelize[t](seq: seq[T], numslices: Int = defaultparallelism) (implicit arg0: classtag[T]): rdd[T]

Distribute a local Scala collection to form an RDD.

Format a native Scala collection as an rdd.

Note

Parallelize acts lazily. If seq is a mutable collection and was altered after the call to Parallelize and before the first action on the RDD, the Resultant RDD would reflect the modified collection. Pass a copy of the argument to avoid this.

Attention

Parallelize is a lazy action function. If the parameter seq is a variable collection, and the period after the call to Parallelize but before an action on the RDD is modified, the resulting RDD will reflect the modified collection, causing the result to be Unpredictable. Therefore, a copy is passed to the parameter seq of this function.

Checkpoint (self)

Mark this RDD for checkpointing. It'll be saved to a file inside the checkpoint directory set with Sparkcontext.setcheckpointdir () and all references to Its parent RDDs would be removed. This function must is called before any job have been executed on the this RDD. It is strongly recommended the This RDD are persisted in memory and otherwise saving it on a file would require recomputation.

Checkpoint (self)

Marks the check point for the current RDD. It is saved as a file in the set of files in the checkpoint directory that is provisioned by the Sparkcontext.setcheckpointdir () method. In short, the current RDD Check point is saved for a file, and this file is in a directory, this directory has a lot of such files, this directory is set by the Sparkcontext.setcheckpointdir () method. And all files referenced from the parent Rdd will be deleted. This function must be called before all jobs and run on this rdd. It is strongly recommended to be stored in memory, otherwise it will need to be recalculated if it is transferred from memory to the file.

Scala:

def setcheckpointdir(directory: String): Unit

Set the directory under which RDDs is going to be checkpointed. The directory must is a HDFS path if running on a cluster.

Set up a directory to allow the RDD to be checkpoint under it. If you are running on a cluster, this directory must be an HDFS path.

Sparkcontext.setcheckpointdir ()