Spark MLib Basic Statistics Summary

1. Summarize Statistics Summary Statistics

MLlibSupports RDD[Vector] column-based summary statistics, which are Statistics  colStats implemented by invoking methods.

colStatsReturns an MultivariateStatisticalSummary  object that contains the column's maximum, minimum, mean, variance, and so on.

Import Org.apache.spark.mllib.linalg.Vectorimport Org.apache.spark.mllib.stat. {multivariatestatisticalsummary, statistics}val observations:rdd[vector] = ...       Define an RDD of Vectors
Compute Column Summary Statistics.val summary:multivariatestatisticalsummary = Statistics.colstats (observations) println (Summary.mean)                     //A dense vector containing the mean value for each columnprintln (summary.variance)                 //COL Umn-wise varianceprintln (Summary.numnonzeros)              //number of Nonzeros in each column

2. Relevance

Spark MLib Basic Statistics Summary