Spark RDD Properties

This article introduces the properties of spark RDD

1. A list to store the preferred locations for accessing each Partition. For an HDFS file, this list stores the location of the block where each Partition is located. According to the concept of "mobile data is not as good as mobile computing", when scheduling tasks, Spark will allocate computing tasks as much as possible to the storage location of the data blocks it processes.

2. Save the function of calculating each partition. This calculation method will be applied to each data block. The calculation of RDD in Spark is in units of shards. Each RDD will implement the compute function to achieve this purpose. The compute function will compound the iterator without saving the result of each calculation.

3. Dependencies between RDDs. Each conversion of RDD will generate a new RDD, so there will be a pipeline-like dependency between RDDs. When part of the partition data is lost, Spark can recalculate the lost partition data through this dependency instead of recalculating all RDD partitions.

4. RDD's partition function (Partitioner), one is HashPartitioner based on hash, and the other is RangePartitioner based on range. Only for key-value RDD, there will be Partitioner, non-key-value RDD Parititioner value is None. The Partitioner function not only determines the number of shards in the RDD itself, but also determines the number of shards when the parent RDD Shuffle is output.

5. Partition (Partition), that is, the basic unit of the data set. For RDD, each shard is processed by a computing task and determines the granularity of parallel computing. The user can specify the number of RDD fragments when creating the RDD. If not specified, the default value will be used. The default value is the number of CPU Cores allocated by the program.