spark1.3.1-based Spark-sql combat-02

Hive Tables

Copy the hive_home/conf/ hive-site.xml file to the spark_home/conf/

When isn't configured by the Hive-site.xml, the context automatically creates metastore_db and warehouse In the current directory.

//SC is an existing sparkcontext.Val SqlContext = New org.Apache.Spark.SQL.Hive.Hivecontext(SC)SqlContext.SQL("CREATE TABLE IF not EXISTS src (key INT, value STRING)")SqlContext.SQL("LOAD DATA LOCAL inpath ' examples/src/main/resources/kv1.txt ' into TABLE src")//Queries is expressed in HiveQLSqlContext.SQL("from SRC SELECT key, value").Collect().foreach(println)

JDBC to other Databases
load MySQL database: Test, table: t_user_new return dataframe

Query data:

Performance Tuning
The Spark Program operates JDBC:





Not to Be continued ~ ~ ~

spark1.3.1-based Spark-sql combat-02