The mutual conversion between 2.sparksql--dataframes and Rdds

Spark SQL supports two ways to convert Rdds to Dataframesuse reflection to get the schema within the RDDusing this reflection-based approach makes the code more concise and effective when the schema of the class is known. specifying schemas through programming interfacescreating the RDD schema from the Spark SQL interface makes the code verbose. The advantage of this approach is that the schema can be generated dynamically when the data columns and the type of columns are known at run time.

The original text is discussed with the author: http://www.cnblogs.com/intsmaze/p/6613755.html

: Intsmaze

using reflection to get Schema (inferring the schema using Reflection)

ImportOrg.apache.spark.sql. {dataframereader, SqlContext}ImportOrg.apache.spark. {sparkconf, sparkcontext}object Inferringschema {def main (args:array[string]) {//Create a sparkconf () and set the app nameVal conf =NewSparkconf (). Setappname ("Sql-intsmaze")    //SqlContext to rely on SparkcontextVal sc =Newsparkcontext (conf)//Create SqlContextVal SqlContext =NewSqlContext (SC)//Create an RDD from a specified addressVal Linerdd = Sc.textfile ("hdfs://192.168.19.131:9000/person.tzt"). Map (_.split (","))    //Create case Class//associating an RDD with a case classVal Personrdd = linerdd.map (x = = person (x (0). ToInt, X (1), X (2). ToInt))//Import implicit conversions, if not import cannot convert Rdd to Dataframe//convert Rdd to Dataframe    Importsqlcontext.implicits._ Val persondf=PERSONRDD.TODF//Registration FormPersondf.registertemptable ("Intsmaze")    //Incoming SQLVal df = Sqlcontext.sql ("SELECT * from Intsmaze ORDER BY age desc limit 2")    //stores the result as JSON to the specified locationDf.write.json ("Hdfs://192.168.19.131:9000/personresult")    //Stop Spark Contextsc.stop ()}}//Case class must be put outside . Case classPerson (Id:int, name:string, Age:int)

There is no need to import sqlcontext.implicits._ in the spark shell because the spark shell is automatically imported by default. package submission to yarn cluster:

Specify schema via programming interface (programmatically specifying the schema)

When JavaBean cannot be pre-defined, programming dataframe is divided into three steps:

Create a row-formatted RDD from the original RDD.

Creates a structtype that matches the rows structure in the RDD and creates a schema that represents the RDD through the Structtype.

The Dataframe is created with the Createdataframe method provided by SqlContext, and the method parameter is the RDD schema.

ImportOrg.apache.spark.sql. {Row, SqlContext}Importorg.apache.spark.sql.types._ImportOrg.apache.spark. {sparkcontext, sparkconf}object Specifyingschema {def main (args:array[string]) {//Create a sparkconf () and set the app nameVal conf =NewSparkconf (). Setappname ("Sql-intsmaze")    //SqlContext to rely on SparkcontextVal sc =Newsparkcontext (conf)//Create SqlContextVal SqlContext =NewSqlContext (SC)//Create an RDD from a specified addressVal Personrdd = sc.textfile (args (0)). Map (_.split (","))    //specify schema for each field directly through StructtypeVal schema =Structtype (List (Structfield ("id", Integertype,true), Structfield ("Name", StringType,true), Structfield ("Age", Integertype,true)      )    )    //map the Rdd to RowrddVal rowrdd = Personrdd.map (p = = Row (P (0). ToInt, P (1). Trim, P (2). ToInt))//applying schema information to RowrddVal Persondataframe =Sqlcontext.createdataframe (Rowrdd, schema)//Registration FormPersondataframe.registertemptable ("Intsmaze")    //Execute SQLVal df = Sqlcontext.sql ("SELECT * from Intsmaze ORDER BY age desc")    //stores the result as JSON to the specified locationDf.write.json (args (1))    //Stop Spark Contextsc.stop ()}}

make the program into a jar package, upload to the spark cluster, submit the Spark task

/home/hadoop/app/spark/bin/spark-submit--class specifyingschema \
--master yarn \
--deploy-mode cluster \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 2 \
--queue default \
/home/hadoop/sparksql-1.0-snapshot.jar \
Hdfs://192.168.19.131:9000/person.txt Hdfs://192.168.19.131:9000/intsmazeresult

/home/hadoop/app/spark/bin/spark-submit--class specifyingschema--master yarn--deploy-mode client--driver-memory 512m--executor-memory 512m--executor-cores 2--queue default/home/hadoop/sparksql-1.0-snapshot.jar hdfs:// 192.168.19.131:9000/person.txt Hdfs://192.168.19.131:9000/intsmazeresult

adding Spark SQL dependencies to the pom.xml of the Maven project

<Dependency> 　　 <groupId>Org.apache.spark</groupId> 　　 <Artifactid>spark-sql_2.10</Artifactid> 　　 <version>1.6.2</version></Dependency>

The mutual conversion between 2.sparksql--dataframes and Rdds