5 Important Internal Properties of Spark RDD

1. One Partition (Partition)

The basic unit of the data set. For RDD, each shard is processed by a computing task and determines the granularity of parallel computing. The user can specify the number of RDD fragments when creating the RDD. If not specified, the default value will be used. The default value is the number of CPU Cores allocated by the program.

2. A function to calculate each partition

The calculation of RDD in Spark is in units of shards, and each RDD will implement a compute function to achieve this purpose. The compute function will compound the iterator without saving the result of each calculation.

3. Dependencies between RDDs

Each conversion of RDD will generate a new RDD, so there will be a pipeline-like dependency between RDDs. When part of the partition data is lost, Spark can recalculate the lost partition data through this dependency instead of recalculating all RDD partitions.

4. A Partitioner

That is RDD's sharding function. Two types of sharding functions are currently implemented in Spark, one is a hash-based HashPartitioner, and the other is a range-based RangePartitioner. Only for key-value RDD, there will be Partitioner, non-key-value RDD Parititioner value is None. The Partitioner function not only determines the number of shards in the RDD itself, but also determines the number of shards when the parent RDD Shuffle is output.

5. A list
Store and access the preferred location of each Partition (preferred location). For an HDFS file, this list stores the location of the block where each Partition is located. According to the concept of "mobile data is not as good as mobile computing", when scheduling tasks, Spark will allocate computing tasks as much as possible to the storage location of the data blocks it processes.