Job Logic Execution diagram

General logical Plan

The typical Job logic execution diagram shows the following four steps to get the final execution result:

• The initial RDD is created from the data source (which can be a local file, memory data structure, hdfs,hbase, etc.) to read the data. The example in the previous chapter parallelize () is equivalent to Createrdd ().
• A series of transformation () operations are performed on the RDD, and each transformation () produces one or more rdd[t with different types of T]. T can be a basic type or data structure within Scala, not limited to (K, V). But if it is (k, V), K cannot be a complex type such as Array (because it is difficult to define the partition function on a complex type).
• The action () operation is performed on the final final RDD, and each partition evaluates to produce the result.
• The result is echoed to the driver end for the final F (List[result]) calculation. The count () in the example actually contains the action () and SUM () two-step calculations.

The RDD can be cache to memory or checkpoint to disk. The number of partition in the RDD is not fixed and is usually set by the user. The partition dependency between Rdd and Rdd can be not 1 to 1, as there are 1 to 1 relationships and many-to-many relationships.

Generation of logical execution graphs

After understanding the Job's logical execution diagram, the writing program will form a data dependency graph similar to the above in the brain. However, the actual number of RDD generated is often more than we think about.

To solve the problem of logical execution diagram generation, we need to solve the problems:

• How do I generate an rdd, and which rdd should I generate?
• How do I build a dependency between RDD?

1. How do I generate an rdd and which rdd should I generate?

The initial idea to solve this problem is to have each transformation () method return (new) an RDD. This is basically true, but some transformation () are more complex and contain multiple sub-transformation (), resulting in multiple RDD. That's why the actual number of RDD is more than we think .

How do I calculate the data in each RDD? The logical execution diagram is actually computing chain, so where is the computational logic of transformation () being perform? Each RDD has the compute () method, which is responsible for receiving the calculation logic of Input records,perform transformation () from the previous Rdd or data source, and then outputting the records.

What RDD is generated is related to the computational Logic of Transformation (), and the following is a discussion of some typical transformation () and the Rdd it creates. The official web has explained the meaning of each transformation. Iterator (split) means the foreach record in the partition. There is a lot of space, because those transformation () are more complex and will produce multiple RDD, which will be shown in the next section.

Transformation	Generated RDDs	Compute ()
Map (func)	Mappedrdd	Iterator (split). Map (f)
Filter (func)	Filteredrdd	Iterator (split). Filter (f)
FlatMap (func)	Flatmappedrdd	Iterator (split). FlatMap (f)
Mappartitions (func)	Mappartitionsrdd	F (iterator (split))
Mappartitionswithindex (func)	Mappartitionsrdd	F (Split.index, iterator (split))
Sample (withreplacement, fraction, Seed)	Partitionwisesampledrdd	Poissonsampler.sample (iterator (split)) Bernoullisampler.sample (iterator (split))
Pipe (command, [Envvars])	Pipedrdd
Union (Otherdataset)
Intersection (Otherdataset)
Distinct ([numtasks]))
Groupbykey ([Numtasks])
Reducebykey (func, [Numtasks])
Sortbykey ([ascending], [numtasks])
Join (Otherdataset, [numtasks])
Cogroup (Otherdataset, [numtasks])
Cartesian (Otherdataset)
COALESCE (Numpartitions)
Repartition (Numpartitions)

2. How do I establish a connection between RDD?

The data dependency problem between RDD actually consists of three parts:

• The dependency of the RDD itself. Is the RDD to be generated (represented later in Rdd X) dependent on a parent RDD or multiple parent RDDs?
• How many partition will there be in RDD x?
• What is the dependency between RDD X and partition in its parent RDDs? Is it dependent on one or more partition in the parent RDD?

The first problem can be solved naturally, for example x = rdda.transformation(rddb) (e.g., x = A.join (b)) means that Rdd X also relies on Rdd A and Rdd B.

The number of partition in the second question is generally specified by the user and is not specified max(numPartitions[parent RDD 1], .., numPartitions[parent RDD n]) .

The third problem is more complicated. The semantics of this transformation () need to be considered, and the dependencies of different transformation () are different. For example, map () is 1:1, and each partition in Shuffledrdd in the Groupbykey () logic execution diagram relies on all partition in the parent RDD, and more complex cases.

Consider the third question again, and each partition in Rdd X can rely on one or more partition in the parent RDD. And this dependency can be either completely dependent or partially dependent. Partial dependency refers to the fact that part of the data in a partition in the parent RDD is related to one partition in the Rdd x, and the other data is related to another partition in the Rdd x. Shows complete dependencies and partial dependencies.

The first three are completely dependent, and the partition in the RDD x are fully related to the partition/partitions in the parent rdd. The last one is a partial dependency, and the partition in the RDD x is only related to the partition part of the parent Rdd, and the other data is related to the other partition in Rdd x.

In Spark, full reliance is called narrowdependency, and part of the dependency is called shuffledependency. In fact Shuffledependency is the same as shuffle data in MapReduce (Mapper will partition its output, and each reducer will own mapper in all partiti outputs On Via HTTP fetch).

• The first type of 1:1 is known as onetoonedependency.
• The second type of N:1 is known as N:1 narrowdependency.
• The third case of N-narrowdependency is known as N. The full dependency that does not belong to the first two cases falls into this category.
• The fourth type is called Shuffledependency.

For Narrowdependency, the partitoin i in the specific Rdd x relies on a partition or multiple partitions in the Parrent Rdd, which is determined by the RDD x getParents(partition i) (some examples in detail). There is also a rangedependency full dependency, but the dependency is currently only used in Unionrdd, as described below.

So, summarize the dependencies between partition as follows:

• Narrowdependency (indicated by a solid black line or a dashed black arrow)
  • Onetoonedependency (1:1)
  • Narrowdependency (N:1)
  • Narrowdependency (n/a)
  • Rangedependency (only used in Unionrdd)
• Shuffledependency (indicated with red arrows)

The reason to divide narrowdependency and shuffledependency is to generate a physical execution diagram, which is described in the next chapter.

It is important to note that the third type of Narrowdependency (n: s) rarely occurs between two RDD. Because if the partition in the parent RDD is also dependent on multiple partitions in the child Rdd, the last dependent graph generated is often the same as shuffledependency. Just for the partition in the parent RDD one is completely dependent, one is partially dependent, and the number of arrows is not small. So the narrowdependency that Spark defines is actually "each partition of the parent Rdd was used by at the very one partition of the child Rdd", which is Only Onetoonedependency (1:1) and narrowdependency (N:1) are the two cases. However, your own design of the wonderful RDD can actually show the Narrowdependency (n-to) situation. The comparison described here, in fact, understand the following a few typical RDD dependency can be.

How do I calculate the data in RDD x (Records)? Shows the Onetoonedependency data dependency, although between partition and partition is 1:1, but does not mean that the calculation records is also read a record to calculate a record. The difference between the top and bottom two pattern is similar to the following two programs:

Code1 of ITER.F ()

= {5} for(< array. length; I+ +) F (array[i])    

Code2 of F (ITER)

= {5}f (array)

3. Some typical transformation () calculation processes and data dependency graphs are given.

1) Union (OTHERRDD)

Union () simply merges the two RDD together without changing the data inside the partition. Rangedependency is actually 1:1, just to access the partition in the RDD after the Union (), preserving the range boundary of the original RDD.

2) Groupbykey (numpartitions)

The previous chapter has introduced the Groupbykey data dependence, here is warm so know new bar.

Groupbykey () only need to aggregate the same records as Key, a simple shuffle process can be done. The compute () in SHUFFLEDRDD is only responsible for the fetch of data belonging to each partition, followed by the Mappartitions () operation (previously shown in onetoonedependency) for aggregate, Into Mappartitionsrdd, where Groupbykey () has ended. Finally, in order to unify the return value interface, the arraybuffer[] data structure in value is abstracted into iterable[].

Groupbykey () does not combine on the map side, because the map side combine will only omit the space that the partition duplicates the key occupies, when repeats the key is very much, may consider to open combine.

The ArrayBuffer here should actually be Compactbuffer,an append-only buffer similar to ArrayBuffer and more memory-efficient for small buf Fers.

Parallelcollectionrdd is the most basic rdd, and the RDD that is created directly from the local data structure belongs to this type, such as

= Sc.parallelize (List (3) 

The generated pairs is parallelcollectionrdd.

2) Reduceybykey (func, Numpartitions)

Reduceybykey () is equivalent to the traditional MapReduce, and the entire data stream is essentially the same as the data flow in Hadoop. Reduceybykey () opens combine () By default on the map side, so combine by mappartitions operation prior to shuffle, Mappartitionsrdd, then shuffle SHUFFL Edrdd, and then reduce (through the aggregate + mappartitions () operation to achieve) get mappartitionsrdd.

3) distinct (numpartitions)

The distinct () feature is all the duplicate data in the Deduplicate RDD. Since duplicate data may be scattered in different partition, it is necessary to shuffle to aggregate and then to re-weigh. However, the shuffle requires that the data type be <K, V> . If the original data has only Key (for example, the record has only one integer), then it needs to be added <K, null> . This supplemental process is done by the map () operation, generating Mappedrdd. Then call the above Reducebykey () to perform the shuffle, combine on the map side, then reduce further to the weight, generate Mappartitionsrdd. Finally, it will <K, null> revert to K and still be completed by map (), generating Mappedrdd. The blue part is called the Reducebykey ().

4) Cogroup (Otherrdd, numpartitions)

Unlike Groupbykey (), Cogroup () aggregate two or more two RDD. So the relationship between Cogroupedrdd and Rdd A and Rdd B must be shuffledependency? Is there a onetoonedependency?

The first thing to be clear is that COGROUPEDRDD exists several partition can be set directly by the user, regardless of rdd a and Rdd B. However, if the number of partition in Cogroupedrdd is different from the number of partition in the RDD A/b, then there is no 1:1 relationship.

Again, the Cogroup () calculation results in Cogroupedrdd which partition is determined by the user-set Partitioner (the default is Hashpartitioner). So it can be introduced: even if the number of partition in the Rdd A/b is the same as in Cogroupedrdd, if the partitioner in the Rdd A/b is not the same as in Cogroupedrdd, there is no 1:1 relationship. For example, in the example, RDD A is rangepartitioner,b Hashpartitioner,cogroupedrdd is also Rangepartitioner and the number of partition is the same as a. Then naturally, each partition in a records can be sent directly to the corresponding partition in Cogroupedrdd. The records in RDD B must be divided and shuffle again before entering the corresponding partition.

Finally, through the above analysis, for two or more than two Rdd aggregation, and only if the Partitioner category and the partition number in the aggregated rdd are the same as the previous RDD, it will be a 1:1 relationship with the previous rdd. Otherwise, it can only be shuffledependency. The code corresponding to this algorithm can be found in the CoGroupedRDD.getDependencies() , although it is difficult to understand.

How does the Spark code indicate that partition in Cogroupedrdd relies on partitions in multiple parent RDDs?

First, put all the RDD that Cogroupedrdd relies on into the array Rdds[rdd]. Again, foreach I, If Cogroupedrdd and Rdds (i) correspond to the RDD is onetoonedependency relationship, then dependecy[i] = new Onetoonedependency (RDD), otherwise = New Shuffledependency (RDD). Finally, return an array of dependencies with each parent RDD deps[dependency].

The getparents (partition ID) in the Dependency class is responsible for giving the Dependency in the parent RDD that a partition depends on in the partitions:list[int].

GetPartitions () is responsible for giving the number of partition in the RDD and how each partition is serialized.

5) Intersection (OTHERRDD)

The intersection () feature extracts common data from Rdd A and Rdd B. First use map () to convert rdd[t] to rdd[(T, NULL)], where T is not a collection type such as Array. Next, proceed to A.cogroup (b), the blue part is the same as the previous cogroup (). Then filter () filters out [ITER (Groupa ()), ITER (GroupB ())] Groupa or GroupB empty records, get Filteredrdd. Finally, use keys () to keep key only, and get Mappedrdd.

6) Join (Otherrdd, numpartitions)

Join () aggregates two rdd[(K, V)] According to the join method in SQL. Similar to intersection (), Cogroup () is first performed, the <K, (Iterable[V1], Iterable[V2])> Mappedvaluesrdd of the type is obtained, and then the Cartesian set of Iterable[v1] and Iterable[v2] is made, and the set is flat ().

Here are two example, the first example Rdd 1 and Rdd 2 are divided using Rangepartitioner, and Cogroupedrdd uses Hashpartitioner, unlike the Rdd 1/2, so it is Sh Uffledependency. In the second example, the RDD 1 uses Hashpartitioner to divide its key in advance to get three partition, which is consistent with the Hashpartitioner (3) used by Cogroupedrdd, so data dependency is 1:1. If the RDD 2 is also using Hashpartitioner to divide its key, get three partition, then join () does not exist shuffledependency, the join () becomes hashjoin ().

7) Sortbykey (Ascending, numpartitions)

Sortbykey () Sorts the records in rdd[(K, V)] by key, ascending = True for ascending, and false for descending. Currently Sortbykey () data dependency is very simple, first use shuffle to gather records together (put into the corresponding partition), and then partition all records by key sort, and finally get Mapparti The records in the Tionsrdd is orderly.

Currently Sortbykey () uses an Array to save all the records in the partition, and then sorts.

8) Cartesian (OTHERRDD)

Cartesian a Cartesian set of two RDD, the number of partition generated in Cartesianrdd = Partitionnum (Rdd a) * Partitionnum (Rdd b).

The dependencies here are not the same as in the previous one, and each of the partition in Cartesianrdd relies on two parent RDD, each of which relies entirely on one partition in Rdd A and relies entirely on another par in Rdd b. Tition. There are no Red arrows, because all dependencies are narrowdependency.

Cartesianrdd.getdependencies () returns Rdds[rdd A, RDD b]. Partiton I in Cartesianrdd is dependent on (RDD a). List (I/NUMPARTITIONSINRDDB) and (RDD B). List (i% numpartitionsinrddb).

9) Coalesce (numpartitions, shuffle = False)

COALESCE () can adjust the number of partition of the parent RDD, such as from 5 to 3, or from 5 to 10. It is important to note that when Shuffle = false, it is not possible to increase the number of partition (cannot be changed from 5 to 10).

The core issue of coalesce () is how to establish the partition relationship between partition in Coalescedrdd and its parent RDD.

• COALESCE (Shuffle = false), the problem becomes the parent RDD which partition can be merged together because the shuffle cannot be performed. In addition to considering the number of elements in partition, the merging factors should also consider the problems of locality and balance. So, Spark designed a very complex algorithm to solve the problem (the algorithm part I haven't delved into). Attention Example: a.coalesce(3, shuffle = false) shows the narrowdependency of N:1.
• COALESCE (Shuffle = true), the problem becomes how to divide all records in the RDD into N partition, because shuffle can be done. Very simply, in each partition, append a key,key increment to each record, so that after the hash (key), the key can be evenly distributed to different partition, similar to the Round-robin algorithm. In the second example, each element in RDD A is preceded by an incremented key (such as 1 in the second partition of Mappartitionsrdd (1, 3). In each partition, the key in the first element (key, Value) (new Random(index)).nextInt(numPartitions) is computed, index is the partition, and Numpartitions is the number of partition in Coalescedrdd 。 The next element of the key is incremented, and then shuffle Shuffledrdd can be divided into records, and then through the complex algorithm to establish the data between Shuffledrdd and Coalescedrdd, finally filter out key, get coal ESCE results after the Mappedrdd.

Repartition (Numpartitions)

Equivalent to COALESCE (numpartitions, shuffle = True)

Primitive Transformation ()

Combinebykey ()

Having analyzed so many of the RDD's logical execution graphs, are there any similarities between them? If so, how are they designed and implemented?

A careful analysis of the RDD's logical execution diagram reveals that the record requirement in the RDD on the left side of the shuffledependency is <key, value> type, after shuffledependency, containing the same key records are aggregate together and perform different computational logic on the aggregated records. When actually executed (later chapters will specifically refer to) many transformation () such as Groupbykey (), Reducebykey () is the edge aggregate data side to perform computational logic, so the common thing is aggregate at the same time compute ()。 Spark uses Combinebykey () to implement the underlying operation of this aggregate + compute ().

Combinebykey () is defined as follows:

 Defcombinebykey[c] (createcombiner: v => C, mergevalue: (c, v) => c, mergecombiners: (c, C) => c, partitioner: partitioner, mapsidecombine: boolean = true, serializer: serializer = null) rdd[(k, c)]    

There are mainly three parameters Createcombiner,mergevalue and mergecombiners. Simply explain the meanings of these three functions and Combinebykey () and note their types:

Assuming a set of <k with the same K, V> Records is flowing to Combinebykey (), Createcombiner initializes the value of the first record to C (for example, C = value), and then from the second R Ecord start with a record that uses Mergevalue (c, record.value) to update C, for example, to sum all the values of these records, use C = C + Record.value. Wait until records is all Mergevalue () and get the result c. Suppose there is a set of records (key is the same as the key in the previous group), and Combinebykey () uses the previous method to calculate the C ' continuously. Now if the two groups are required to records the result of the total Combinebykey (), then the final c = Mergecombiners (c, C ') can be used to calculate.

Discussion

At this point, we discussed how to generate a logical execution diagram of the job, which is the complex computational logic and data dependencies behind the seemingly simple API of Spark.

The entire job will produce which RDD is determined by transformation () semantics. Some transformation (), such as Cogroup (), are used by many other operations.

The dependency of the RDD itself is determined by the semantics of each RDD generated by transformation (). such as Cogroupedrdd relies on all RDDs participating in Cogroup ().

The partition dependencies in the RDD are divided into narrowdependency and shuffledependency. The former is completely dependent and the latter is partly dependent. Narrowdependency also contains a variety of situations, only the number of partition and two RDD and Partitioner are the same, will appear narrowdependency.

From the point of view of data processing logic, MapReduce corresponds to the map () + Reducebykey () in Spark, but strictly speaking, the reduce () in MapReduce is more powerful than Reducebykey (), and the detailed differences will be shuff Further discussion in the Le details chapter.

Job Logic Execution diagram