Sparksql using the Thrift JDBC server

Thrift JDBC Server Description

Thrift JDBC Server uses the HIVESERVER2 implementation of HIVE0.12. Ability to use Spark or hive0.12 versions of Beeline scripts to interact with JDBC server. The Thrift JDBC server default listening port is 10000.

Before using Thrift JDBC Server, you need to be aware of:

1, copy the Hive-site.xml configuration file to the $spark_home/conf directory;

2. Need to add JDBC-driven jar packages to Spark_classpath in $spark_home/conf/spark-env.sh

Export Spark_classpath= $SPARK _classpath:/home/hadoop/software/mysql-connector-java-5.1.27-bin.jar

Thrift JDBC Server command use Help:

CD $SPARK _home/sbinstart-thriftserver. SH --help

Usage:./sbin/start-thriftserver [Options] [Thrift Server Options]spark assembly have been built with Hive, including DataNucleus jars on C Lasspathoptions:--master Master_url spark://host:port, mesos://host:port, yarn, or Local.--deploy-mode Deploy_mode Whether to launch the driver program locally ("Client") or on one of the worker machines inside the cluster ("Cluster") (default:client). --class class_name Your Application'S main class (for Java/scala apps).--name Name A name of your application. --jars Jars comma-separated list of local jars to include on the driver and executor classpaths. --py-files py_files comma-separated List of.Zip,. Egg, or. py files to place on the PYTHONPATH forPython Apps. --files files comma-separated List of files to be placedinchThe working directory of each executor. --conf prop=VALUE arbitrary Spark configuration property. --properties-fileFILE Path to afileFromwhichTo load extra properties. If not specified, the this would look forconf/spark-defaults.conf.--driver-memory MEM Memory fordriver (e.g. 1000M, 2G) (default:512m). --driver-java-options Extra Java options to pass to the driver. --driver-library-Path Extra Library path entries to pass to the driver. --driver-class-path Extra class path entries to pass to the driver. Note that jars added with--jars is automatically includedinchThe classpath. --executor-memory MEM memory per executor (e.g. 1000M, 2G) (default:1g). --help,-h Show This help message and exit--verbose,-v Print additional debug output Spark standalone with cluster deploy mode only:--driver-cores NUM Cores forDriver (Default:1). --supervise If Given, restarts the driver on failure. Spark Standalone and Mesos only:--total-executor-cores NUM Total Cores forAll executors. YARN-Only :--executor-cores NUM number of cores per executor (Default:1). --queue queue_name the YARN queue to submit to (Default:"default"). --num-executors num number of executors to launch (Default:2). --Archives Archives Comma separated list of archives to BES extracted into the Worki ng directory of each executor. Thrift Server options:--hiveconf <property=value> Use value forGiven property

Master's description is consistent with the spark SQL CLI

Beeline command Use Help:

CD $SPARK _home/--help

Usage:java Org.apache.hive.cli.beeline.BeeLine-U <database url>The JDBC URL to connect to-N <username>The username to connect as-P <password>The password to connect as-D <driver class>The driver class to use-E <query>query that should be executed-F <file> ScriptfileThat's should be executed--color=[true/false] Control whether color is used forDisplay--showheader=[true/false] Show Column namesinchQuery Results--headerinterval=rows; The interval betweenwhichHeades is displayed--fastconnect=[true/false] Skip Building Table/column List fortab-Completion--autocommit=[true/false] enable/Disable automatic transaction commit--verbose=[true/false] Show verbose error messages and debugInfo--showwarnings=[true/false] Display Connection warnings--shownestederrs=[true/false] Display nested errors--numberformat=[pattern] format numbers using DecimalFormat pattern--force=[true/false] Continue running script even after errors--maxwidth=MAXWIDTH The maximum width of the terminal--maxcolumnwidth=maxcolwidth The maximum width to use when displaying columns--silent=[true/false] Be MoreSilent--autosave=[true/false] Automatically save preferences--OUTPUTFORMAT=[TABLE/VERTICAL/CSV/TSV] Format mode forresult Display--isolation=Level Set the transaction isolation level--HELP Display this message

Thrift JDBC server/beeline Boot

start Thrift JDBC Server: Default port is 10000

CD $SPARK _home/sbinstart-thriftserver. SH

How do I modify the default listening port number for thrift JDBC server? With the help of--hiveconf

Start-thriftserver. SH  --hiveconf hive.server2.thrift.port=14000

HiveServer2 clients details see: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients

Start Beeline

CD $SPARK _home/-u jdbc:hive2://hadoop000:10000/default-n Hadoop

SQL Script Test

SELECT  from WHERE = -  + Ten ; SELECT Count (*fromgroupbyorderbydesc10 ;

Sparksql using Thrift JDBC Server