Spark Source Analysis-scheduler Module

RDD dependencies and classification of the stage

In Spark, each is a representation of a RDD dataset in a certain state, and this state is likely to be transformed from the previous state, so in other words it RDD may RDD(s) have dependencies on the previous one. Depending on the dependencies, you can RDD divide into two different types: Narrow Dependency and Wide Dependency .

• Narrow DependencyRefers to child RDD only a parent RDD(s) fixed number of partition.
• Wide DependencyEach of the child RDD partition is dependent on parent RDD(s) all partition.

Http://jerryshao.me/architecture/2013/04/21/Spark%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%E4%B9%8B-scheduler%E6%A8 %a1%e5%9d%97/

Spark Source Analysis-scheduler Module