Spark hive combined with notes

1. Download Spark Source Code, there is a make-distribution.sh file under the Spark Source directory, modify the parameters inside, so that the compilation can support hive, after modification to execute the file. (Maven must be installed in advance before compilation ).

2. Deploy the compiled Spark Source code on the machine and copy the hive-site.xml in hive/conf to the spark/conf directory. Then you can test it through spark-shell, see: http://www.cnblogs.com/hseagle/p/3758922.html

3. When using hive in Spark, start the hive Server Service, create a step file in the hive/bin directory, and add chmod U + x filename

nohup ./hive --service hiveserver >> hiveserver.log 2>&1 &echo $! > hive-server.pid

4. When writing a spark program in eclipse, add the spark/lib package to spark-examples .... Jar does not need to be imported. If you want to operate hive, you may need to import the corresponding database JDBC driver.

5. when you directly run the spark hive program of Java in eclipse, the hive MetaStore may be incorrect first, but the default MetaStore linked. In this case, various errors may occur. no table tablename, can't fetch table ..., A similar error occurs. My solution is to package the Java program into a jar and copy the exported jar package to the spark/lib directory ., Then, the jar program is executed by referring to the spark-submit script command in Spark/bin. The following is a simplified example of spark/bin/run-examples, after the test, you can perform simple input without parameters:

SCALA_VERSION=2.10FWDIR="$(cd `dirname $0`/..; pwd)"export SPARK_HOME="$FWDIR"export SPARK_EXAMPLES_JAR=$SPARK_HOME/lib/YOUR_EXPORT_JAR_NAME.jarEXAMPLE_MASTER=${MASTER:-"local[*]"}EXAMPLE_CLASS=demo.wrencai.cup.SparkHiveDemo"$FWDIR"/bin/spark-submit   --master $EXAMPLE_MASTER   --class $EXAMPLE_CLASS   --name SparkSubmit_Demo   "$SPARK_EXAMPLES_JAR" \

Note: In the above script, you need to modify the two lines marked in blue and change the red font to the jar package name you export and the class name you want to execute g (Format: yourpacketname. yourclassname)

 

Spark hive combined with notes