Scala-spark Lambda "Goesto" = Analysis

1 ///define a function addnoise, the parameters are rdd,fraction respectively. Where the Rdd is an RDD (Breezedensematrix, Breezedensematrix) tuple. Fraction is a double. Returns an RDD consisting of a (Breezedensematrix, Breezedensematrix) tuple. 2def addnoise (rdd:rdd[(bdm[double], bdm[double])], fraction:double): rdd[(bdm[double], bdm[double]) {3 ///defines the return value temporary storage, which is implemented by each element of the Rdd f=>sth composition4Val addnoise = rdd.map {f = =5 ///F The second part of the data for a Breezedensematrix6Val features =f._27 ///Generates a Breezedensematrix, populated by random numbers8Val A =Bdm.rand[double] (features.rows, Features.cols)9 ///Defines a Breezedensematrix A1, where the element is of type bool. True if the corresponding element in a is greater than fraction. otherwise, false. TenVal A1 = A: >=Fraction One ///Defines an rdd D1, populated by the elements in A1 after the following operation: If the current element is true, then 1.0, otherwise 0.  AVal D1 = a1.data.map {f = =if(f = =true) 1.0Else0.0 } - ///Create a new Breezedensematrix, which is populated by the D1 element corresponding to the features corresponding position.  -Val A2 =NewBDM (features.rows, Features.cols, D1) the ///:* means that each element is multiplied sequentially. Get Breezedensematrix.  -Val features2 = Features:*A2 - ///Return (Breezedensematrix,breezedensematrix) constitutes the RDD. As a function return value, update addnoise.  - (F._1, Features2) +     } - ///Returns the result of the operation as a function return value.  + addnoise A}

This code is written in Scala with the addnoise implementation of the NN algorithm that runs on the spark. Used to complete the denoiseautoencoder random noise addition.

The idea is simple, but it's worth learning to show the map manipulation of spark and the use of the lambda operator in Scala.

The code comes from Sunbow0.

Personal analysis, please correct me if you are wrong.

Scala-spark Lambda "Goesto" = Analysis