spark-1.6.1 Installing and compiling &&sparksql operation Hive

maven:3.3.9

Jdk:java Version "1.8.0_51"

Spark:spark-1.6.1.tgz

scala:2.11.7

If the Scala version is 2.11.x, execute the following script

./dev/change-scala-version.sh 2.11

Spark is compiled by default with Scala's 2.10.5

The compile command is as follows:

mvn-pyarn-phadoop-2.6-dhadoop.version=2.6.0 -phive-phive-thriftserver-dscala-2.11 -DskipTests Clean Package

The red section is the required dependency for Spark-sql link hive, and the specified Scala version

Note: The Hive-site.xml file needs to be placed in the $spark_home/conf directory or the table cannot be found

Using Spark-sql to access hive

Package Com.infra.codelab.spark.hiveimport Org.apache.spark.SparkConfimport Org.apache.spark.SparkContextobject hivetest {val conf = new sparkconf () Val sc = new Sparkcontext (conf) def main (args:array[string]): Unit = {val sq Lcontext = new Org.apache.spark.sql.hive.HiveContext (SC) sqlcontext.sql ("Select line from Filecontent"). Collect (). For Each (println)}}

Submit a task:

Spark-submit--class com.infra.codelab.spark.hive.HiveTest--master Spark://localhost:7077/home/xiaobin/test/spark /wordcount-0.0.1-snapshot.jar

Spark-sql:

Export Spark_classpath= $SPARK _classpath:/home/xiaobin/soft/apache-hive-0.14.0-bin/lib/ Mysql-connector-java-5.1.35.jar

Spark-sql--master spark://xiaobin:7077

Spark-sql> Select COUNT (*) from filecontent;483 time Taken:3. 628 seconds, fetched 1 row (s)

This article is from the "11660039" blog, please be sure to keep this source http://11670039.blog.51cto.com/11660039/1784692

spark-1.6.1 Install compile &&sparksql operation Hive