RDD persistence (Spark) _rdd

RDD Persistence Storagelevel Describe none RDD do not persist disk_only RDD partitions are persisted only on disk disk_only_2 _2, each partition is backed up to 2 cluster nodes, others ditto Memory_ Only the default persistence policy. Rdd is deserialized as a Java object and persisted into the JVM virtual machine memory, and some RDD partition data cannot be persisted if the memory cannot accommodate all RDD partitions. That is, cannot be hit (cached), when these RDD partitions are used again, the memory_only_2 _2 will need to be recalculated, and each partition will be backed up to 2 cluster nodes, the other ibid memory_only_ser and memory_only The only difference: serializing Rdd as a Java object (an array of bytes per partition) is persisted to the JVM virtual machine memory. Pros and Cons: reduce memory overhead, but need to deserialize, consume CPU memory_only_ser_2 _2, back up each partition to 2 cluster nodes, and other ibid. memory_and_disk to persist Rdd as Java objects after deserialization. If the memory does not fully accommodate the RDD data, the remainder is persisted to disk. Subsequent read memory_and_disk_2 _2 on demand from memory and disk, with each partition backed up to 2 cluster nodes, and the other ibid memory_and_disk_ser similar to Memory_only_ser, Memory-less RDD partition data is persisted to disk, avoids, avoids again when need to recalculate memory_and_disk_ser_2 _2, will backup each partition to 2 cluster nodes, other ibid off_heap off_heap parameter is persisted to the distributed memory file system by default. The memory_only_ser,off_heap reduces the cost of garbage collection, making executors smaller and shared memory pools. This is attractive in a high concurrency environment. In addition, because the RDD resides in the Tachyon, the executor crash does not cause data loss, and in this mode, the memory in Tachyon is discarded, so tachyon does not attempt to reconstruct blocks that are purged from memory. References RDD Persistence Authority API Spark Performance Optimization Guide-Basic article