Apache Spark Rdd's Rdd checkpoint

Check points for Rdd

The RDD cache can be saved to memory, local file system, or Tachyon after the first calculation is completed. With caching, spark avoids repetitive computations on the RDD and can greatly increase the computational speed. However, if the cache is missing, it needs to be recalculated. If the calculations are particularly complex or time-consuming, the impact of cache loss on the entire job cannot be overlooked. To avoid the overhead of cache loss recalculation, Spark introduces a checkpoint mechanism.

The cache is written directly to the user-defined storage level (the storage level defines the media for cache storage, now supports memory, local file system, and Tachyon) to write to different media at the end of the calculation. And the checkpoint is different, it is after the calculation is completed, re-establish a job to calculate. To avoid duplication of calculations, it is recommended to cache the RDD first so that the checkpoint can be done quickly.

Apache Spark Rdd's Rdd checkpoint