RDD Basic Conversion Operations (6) –zip, zippartitions

Zip

def Zip[u] (Other:rdd[u]) (implicit arg0:classtag[u]): rdd[(T, U)]

The ZIP function is used to synthesize two RDD groups into an rdd in the form of Key/value, where the partition number of the default two Rdd and the number of elements are the same, otherwise an exception will be thrown.

scala> var rdd1 = Sc.makerdd (1 to 10,2) rdd1:org.apache.spark.rdd.rdd[int] = parallelcollectionrdd[0] at MakeRDD at:2 1 scala> var rdd1 = Sc.makerdd (1 to 5,2) rdd1:org.apache.spark.rdd.rdd[int] = parallelcollectionrdd[1 "at MakeRDD" at : scala> var rdd2 = Sc.makerdd (Seq ("A", "B", "C", "D", "E"), 2) rdd2:org.apache.spark.rdd.rdd[string] = Parallelcollec TIONRDD[2] at Makerdd at:21 scala> rdd1.zip (RDD2). Collect res0:array[(Int, String)] = Array ((1,a), (2,b), (3,c), (4 , D), (5,e)) scala> Rdd2.zip (RDD1). Collect res1:array[(String, Int)] = Array ((a,1), (b,2), (c,3), (d,4), (e,5)) SCA la> var rdd3 = Sc.makerdd (Seq ("A", "B", "C", "D", "E"), 3)Rdd3:org.apache.spark.rdd.rdd[string] = parallelcollectionrdd[5] at Makerdd at:21 scala> rdd1.zip (rdd3). Collect JAV A.lang.illegalargumentexception:can ' t zip RDDs with unequal numbers of partitions//throws an exception if the number of two RDD partitions is different zippartitions

The Zippartitions function adds multiple rdd to the new RDD as partition, which requires the combined RDD to have the same number of partitions, but there is no requirement for the number of elements within each partition.

There are several implementations of this function, which can be divided into three categories: The parameter is an RDD

def Zippartitions[b, V] (Rdd2:rdd[b]) (f: (Iterator[t], iterator[b]) = Iterator[v]) (Implicit arg0:classtag[b], Arg1:classtag[v]): Rdd[v]

def Zippartitions[b, V] (rdd2:rdd[b], Preservespartitioning:boolean) (f: (Iterator[t], iterator[b]) = Iterator[v]) (Implicit arg0:classtag[b], Arg1:classtag[v]): Rdd[v]

These two differences are the parameter preservespartitioning, whether the Partitioner partition information of the parent RDD is preserved

The map method f parameter is an iterator of two rdd.

scala> var rdd1 = Sc.makerdd (1 to 5,2) rdd1:org.apache.spark.rdd.rdd[int] = parallelcollectionrdd[22 "at MakeRDD" at:   scala> var rdd2 = Sc.makerdd (Seq ("A", "B", "C", "D", "E"), 2) rdd2:org.apache.spark.rdd.rdd[string] = PARALLELCOLLECTIONRDD[23] at Makerdd at:21  //RDD1 Two-partition element distribution: scala> rdd1.mappartitionswithindex{| (x,iter) = {| var result = list[string