Spark RDD Dependency Types

According to different conversion operations, the dependency of RDD blood relationship is divided into narrow dependency and broad dependency. Narrow dependency means that each partition of the parent RDD is only used by one partition of the child RDD. Wide dependency means that each partition of the parent RDD is dependent on partitions of multiple child RDDs.

Operations such as map, filter, and union are narrowly dependent, while operations such as groupByKey and reduceByKey are broadly dependent, as shown in Figure 3.

There are two cases for the join operation. If each Partition used in the join operation only joins with a fixed number of Partitions, the join operation is a narrow dependency, and in other cases the join operation is a wide dependency.

Therefore, it can be concluded that narrow dependencies include not only one-to-one narrow dependencies, but also a fixed number of narrow dependencies, that is to say, the Partition that depends on the parent RDD will not change as the RDD data size changes .

1. Narrow Dependence

1) Each partition of the child RDD depends on a constant number of parent partitions (that is, it has nothing to do with the size of the data).

2) One-to-one input and output operators, and the partition structure of the resulting RDD remains unchanged, such as map and flatMap.

3) One-to-one input and output operator, but the result is that the partition structure of RDD has changed, such as union.

4) Select the operator of some elements from the input, such as filter, distinct, subtract, sample.

2. Wide dependence

1) Each partition of the child RDD depends on all parent RDD partitions.

2) Reorganize and reduce a single RDD based on Key, such as groupByKey and reduceByKey.

3) Join and reorganize two RDDs based on Key, such as join.

This dependency design of Spark makes it inherently fault-tolerant and greatly speeds up the execution speed of Spark. RDD remembers how it evolved from other RDDs through blood relationship. When part of the RDD partition data is lost, it can obtain enough information through the blood relationship to recalculate and restore the lost data partition, thereby bringing performance improvements.

Relatively speaking, the narrow-dependency failure recovery is more efficient. It only needs to recalculate the lost partition based on the parent RDD partition, and does not need to recalculate all the partitions of the parent RDD. In terms of wide dependency, if a single node fails, even if only one partition of RDD fails, all partitions of the parent RDD need to be recalculated, which is expensive.

The wide dependency operation is like "shuffling" the records of all partitions in the parent RDD, the data is broken up, and then reorganized in the child RDD.