The aggregate function of Scala learning

Both fold and reduce require the return value type of the function to be the same as the type of rdd we manipulate, but sometimes we do need a different type of return value. eg

When calculating the average, it is necessary to record the count in the facilitation process and the number of elements, which requires us to return a two-tuple. You can use the map operation on the data to transfer the elements to the change element and the 1 two tuple, which is the return type we want. In this way, reduce can be attributed in the form of a two-tuple.

The aggregate function frees us from the restriction that the return value type must be the same as the input RDD type. Similar to fold, when using aggregate, you need the initial value of the type that we expect to return, and then combine the elements in the RDD with a function bar into the accumulator. Given that each node is accumulated locally, it is ultimately necessary to provide a second function to merge the accumulator 22. eg

1Val z = sc. parallelize (List (1,2,3,4,5,6) ,2)2Val result = Z.aggregate ((0,0) ) (//Initial value3(acc,value) = (acc._1+value,acc._2+1),//accumulator (tuple cumulative tuple result, Rdd single element value) = = (Tuple cumulative result +rdd single element, tuple cumulative Count + 1)4(ACC1,ACC2) = (acc1._1+acc2._1,acc1._2+acc2._2)//combine Merge function Merge tuple cumulative result5)
6 val avg = result._1/result._2.todouble

The aggregate function of Scala learning