Spark Lineage (Descent)

The use of memory to speed up data loading, in many other In-memory class database or cache class system is also implemented, Spark's main difference is that it handles the distributed computing environment of data fault tolerance (node effectiveness/data loss) of the scheme used. To ensure the robustness of the data in the RDD, the RDD data set remembers how it evolved from other RDD through the so-called bloodline relationship (lineage). The lineage of the Rdd records coarse-grained specific data conversion (transformation) operations (filter, map, join etc) compared to other systems with fine-grained memory data update level backup or log mechanism. Behavior. When part of this RDD partition data is lost, it can get enough information through lineage to re-compute and recover the lost data partition. This coarse-grained data model limits the use of Spark, but it also brings performance improvements over fine-grained data models.

The RDD is divided into two kinds of Narrow Dependencies and Wide Dependencies in lineage dependence to solve the data fault-tolerant efficiency.

Narrow Dependencies refers to a partition of a parent rdd that is used by a partition of up to one child rdd, as a partition of a parent RDD corresponds to a partition of a child rdd, or to a partition of more than one parent rdd that corresponds to a sub-rdd. This means that a partition of a parent RDD cannot correspond to multiple partitions of a child rdd.

Wide Dependencies refers to the sub-RDD partition that relies on multiple partitions or all partitions of the parent Rdd, that is, one partition of the parent RDD corresponding to multiple partitions of a child rdd. To the wide Dependencies, the input and output of this calculation on different nodes, the lineage method to the input node intact, while the output node down, by recalculation, in this case, this method of fault tolerance is valid, otherwise invalid, because can not retry, It is necessary to retrace its ancestors to see if it is possible to retry (this is the meaning of lineage, descent), Narrow dependencies The cost of the data is much smaller than wide dependencies's data re-accounting overhead.

As shown in the figure:

Fault tolerant

In the RDD calculation, through checkpoint fault tolerance, do checkpoint there are two ways, one is checkpoint data, one is logging the updates. The user can control which way to implement fault tolerance, by default logging the updates method, by recording tracking all the generated RDD conversion (transformations) is recorded each Rdd lineage (descent) to recalculate the generation of lost partition data.