Two ways to convert Rdd into dataframe in Spark (implemented in Java and Scala, respectively)

One: Prepare the data source

Create a new Student.txt file under the project with the following contents:

1, Zhangsan,  2, Lisi,  3, Wanger,  4, Fangliu,

Two: Realize

Java Edition:

1. First create a new student bean object, implementing the serialization and ToString () method, with the following code:

import java.io.Serializable; @SuppressWarnings ("Serial")   Public classStudent implements Serializable {String sid;      String sname; intSage;  PublicString GetSID () {returnSID; }       Public voidSetsid (String sid) { This. Sid =SID; }       PublicString Getsname () {returnsname; }       Public voidsetsname (String sname) { This. sname =sname; }       Public intGetsage () {returnSage; }       Public voidSetsage (intSage) {           This. Sage =Sage; } @Override PublicString toString () {return "Student [sid="+ SID +", Sname="+ sname +", sage="+ Sage +"]"; }    }  

2. Conversion, the specific code is as follows

import java.util.ArrayList;  Import org.apache.spark.SparkConf;  Import Org.apache.spark.api.java.JavaRDD;  Import Org.apache.spark.sql.Dataset;  Import Org.apache.spark.sql.Row;  Import Org.apache.spark.sql.RowFactory;  Import Org.apache.spark.sql.SaveMode;  Import org.apache.spark.sql.SparkSession;  Import Org.apache.spark.sql.types.DataTypes;  Import Org.apache.spark.sql.types.StructField;    Import Org.apache.spark.sql.types.StructType;  Public classTxttoparquetdemo { Public Static voidMain (string[] args) {sparkconf conf=NewSparkconf (). Setappname ("Txttoparquet"). Setmaster ("Local"); Sparksession Spark=sparksession.builder (). config (conf). Getorcreate (); Reflecttransform (spark);//Java ReflectionDynamictransform (Spark);//Dynamic Conversion    }            /** * Convert through Java reflection * @param spark*/      Private Static voidReflecttransform (Sparksession Spark) {Javardd<String> Source = Spark.read (). Textfile ("StuInfo.txt"). Javardd (); Javardd<Student> Rowrdd = Source.map (Line-{String parts[]= Line.split (","); Student Stu=NewStudent (); Stu.setsid (parts[0]); Stu.setsname (parts[1]); Stu.setsage (integer.valueof (parts[2])); returnStu;                    }); Dataset<Row> df = spark.createdataframe (Rowrdd, Student.class); DF.Select("Sid","sname","Sage"). COALESCE (1). write (). Mode (Savemode.append). Parquet ("Parquet.res"); }      /** * Dynamic conversion * @param spark*/      Private Static voidDynamictransform (Sparksession Spark) {Javardd<String> Source = Spark.read (). Textfile ("StuInfo.txt"). Javardd (); Javardd<Row> Rowrdd = Source.map (Line-{string[] parts= Line.split (","); String Sid= parts[0]; String sname= parts[1]; intSage = Integer.parseint (parts[2]); returnrowfactory.create (SID, Sname, Sage          );                    }); ArrayList<StructField> fields =NewArraylist<structfield>(); Structfield Field=NULL; Field= Datatypes.createstructfield ("Sid", Datatypes.stringtype,true);          Fields.Add (field); Field= Datatypes.createstructfield ("sname", Datatypes.stringtype,true);          Fields.Add (field); Field= Datatypes.createstructfield ("Sage", Datatypes.integertype,true);                    Fields.Add (field); Structtype Schema=datatypes.createstructtype (fields); Dataset<Row> DF =Spark.createdataframe (Rowrdd, schema); Df.coalesce (1). write (). Mode (Savemode.append). Parquet ("parquet.res1"); }      }  

Scala version:

Import org.apache.spark.sql.SparkSession Import org.apache.spark.sql.types.StringType Import Org.apache.spark.sql.types.StructField Import Org.apache.spark.sql.types.StructType Import Org.apache.spark.sql.Row Import Org.apache.spark.sql.types.IntegerTypeObjectRdd2dataset { Case classStudent (Id:int,name:string,age:int) def main (Args:array[string]) {val Spark=sparksession.builder (). Master ("Local"). AppName ("Rdd2dataset"). Getorcreate () Import spark.implicits._ reflectcreate (Spark) Dynamiccreate (Spark)}/** * Convert through Java reflection * @param spark*/    Privatedef reflectcreate (spark:sparksession): unit={Import spark.implicits._ val sturdd=spark.sparkcontext.textfile ("Student2.txt")      //TODF () is an implicit conversionVal Studf=sturdd.map (_.split (","). Map (parts?). Student (Parts (0). Trim.toint,parts (1), Parts (2). trim.toint). TODF ()//studf.select ("id", "name", "Age"). Write.text ("result")//specifying column names for writing to a fileStudf.printschema () Studf.createorreplacetempview ("Student") Val Namedf=spark.sql ("select name from student where Age<20")      //nameDf.write.text ("result")//write query results to a filenamedf.show ()}/** * Dynamic conversion * @param spark*/    Privatedef dynamiccreate (spark:sparksession): unit={val Sturdd=spark.sparkcontext.textfile ("Student.txt") Import spark.implicits._ val schemastring="Id,name,age"Val Fields=schemastring.split (","). Map (FieldName = Structfield (FieldName, stringtype, nullable =true)) Val schema=structtype (Fields) Val Rowrdd=sturdd.map (_.split (","). Map (parts?). Row (Parts (0), Parts (1), Parts (2)) Val studf=Spark.createdataframe (Rowrdd, Schema) Studf.printschema () Val Tmpview=studf.createorreplacetempview ("Student") Val Namedf=spark.sql ("select name from student where Age<20")      //nameDf.write.text ("result")//write query results to a filenamedf.show ()}} 

Note: 1. All of the above code has been tested, and the test environment is spark2.1.0,jdk1.8.

2. This code does not apply to previous versions of spark2.0.

Two ways to convert Rdd into dataframe in Spark (implemented in Java and Scala, respectively)