Spark-RDD persistence

A very important feature of spark is that RDD can be persisted in the memory. When performing a persistence operation, each node will persist the RDD partition of its own operation into the memory, and then use the RDD repeatedly, directly use the memory cache partition. in this case, for a scenario where an RDD executes multiple operations repeatedly, it is necessary to calculate the RDD once, and then directly use the RDD without the need to calculate the RDD multiple times.

Clever Use of RDD persistence can even improve the performance of spark programs by 10 times in some scenarios. RDD persistence is very important for iterative algorithms and fast interactive applications.

To persist an RDD, you only need to call its cache () or persist () method. When this RDD is computed for the first time, it will be directly cached in each node, and spark's persistence mechanism will still be automatically fault-tolerant. If any partition of the persistent RDD is lost, spark automatically recalculates the partition using the transformation operation through its source RDD.

The difference between cache () and persisit () Is that cache () is the one and Simplified Method of persist (), and the no-argument version of persist () is called at the underlying layer of cache, at the same time, persist (memory_only) is called to persist the data to the memory. To clear the cache in the memory, you can use the unpersist () method.

Spark will also perform data persistence during shuffle operations, such as writing data to a disk, mainly to avoid re-computing the entire process when the node fails.

 

package cn.rzlee.spark;import org.apache.spark.SparkConf;import org.apache.spark.api.java.JavaRDD;import org.apache.spark.api.java.JavaSparkContext;/** * @Author ^_^ * @Create 2018/11/3 */public class Persist {    public static void main(String[] args) {        SparkConf conf = new SparkConf().setAppName("Persist").setMaster("local[2]");        JavaSparkContext sc = new JavaSparkContext(conf);        JavaRDD<String> lines = sc.textFile("C:\\Users\\txdyl\\Desktop\\log\\in\\data.txt", 1).cache();        long beginTime = System.currentTimeMillis();        long count = lines.count();        System.out.println(count);        long endTime = System.currentTimeMillis();        System.out.println("cost "+(endTime - beginTime) + "millisecond");        beginTime = System.currentTimeMillis();        count = lines.count();        System.out.println(count);        endTime = System.currentTimeMillis();        System.out.println("cost "+(endTime - beginTime) + "millisecond");        sc.close();    }}

 

Spark-RDD persistence