Spark1.6 Datasets Introduction

Apache Spark provides a powerful API to make it possible for developers to use complex analytics. By introducing Sparksql, developers can use these advanced API interfaces to work with structured data (such as database tables, JSON files), and provide an API for object-oriented use of RDD, and development only needs to invoke related methods to use spark to store and compute data. So what Spark1.6 bring us something? The amount ...

SPARK1.6 provides an API for Datesets, which will be a trend for spark in later releases, just as Dateframe,datesets provides a Catalyst for Spark optimizer As well as analysis optimizations for data field queries and support for faster memory encoding. And the dataset expands the compile-time type-checking mechanism to better check for errors before the program runs.

Datasets is a strongly typed, immutable collection of objects, and the Datasets API core is a new encoder that converts the JVM object to the table structure. Enables it to manipulate serialized data and improve memory utilization. At the same time, when users are using it, they find that the API of the general Rdd is very similar and provides many of the same functional transformations. The following code, which splits the words in the text.

RDDs:

Val lines = Sc.textfile ("/wikipedia"= lines  . FlatMap (_.split (""))  ! = "")

DataSets:

Val lines = SqlContext.read.text ("/wikipedia"= lines  . FlatMap (_.split (""))  ! = "" )

Datasets also supports aggregation operations, such as calculating the number of occurrences of each word:

RDDs:

Val counts = words    . GroupBy (_.tolowercase)    = (w._1, w._2.size))

DataSets:

Val counts = words     . GroupBy (_.tolowercase)    . Count ()

By executing the discovery datasets is much faster than the native Rdd. At the same time, the use of RDD requires the developer to optimize the parallel algorithm, or write the way to achieve datasets effect.

Also, another advantage of the DataSets API is to reduce the amount of memory used. Spark can parse the structured data in datasets and optimize the structure in memory to cache the data in datasets. The original RDD, to save a considerable amount of memory space.

Spark1.6 Datasets Introduction