Deploy hive on Apache hadoop2.6 and store the hive data source in MySQL

The premise of integrating hive is that the Apache Hadoop cluster can start normally.

Hadoop version apach2.6.0 hive version: 1.2.1

1. Install MySQL and give permissions:

1.1: Create hive user and password:

Create user ' hive ' identified by ' 123456 '

1.2: Create the database: creates the databases hive DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;

1.3: Authorization allows remote login: Grant all privileges on * * to ' user ' @ ' hostname ' identified by ' passwd ' with GRANT option;

Flush privileges;

1.4 Copy the MySQL driver to the Hive/lib directory

Mysql-connector-java-5.1.38-bin.jar

2. Hive configuration file in conf directory: hive-env.sh

CP Hive-env.sh.template hive-env.sh

Finally, add the following configuration:

Export java_home=/usr/java/jdk1.7. 0_79export hive_home=/home/hadoop/hive-1.2.1-binexport HADOOP_HOME =/home/hadoop/hadoop-2.6.0

3, Hive-site.xml

Files on 3.1hdfs need to be created beforehand, if permission problems occur, modify the corresponding permissions: Hadoop dfs-chmod-r 777 Hdfs://node17:9000/hive/scratchdir

3.2 The following jar files need to be copied to the Hive/lib directory
Home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar
File:///home/hadoop/hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,
File:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,
File:///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,
File:///home/hadoop/hive-1.2.1-bin/lib/zookeeper-3.4.6.jar,
File:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar

<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://node17:9000/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://node17:9000/hive/scratchdir</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive-1.2.1-bin/logs</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://node15:3306/hive?createDatabaseIfNotExist=true</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>

<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>

<property>
<name>hive.aux.jars.path</name>
<value>file:///home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar, file:///home/hadoop/ Hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,file :///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/ Zookeeper-3.4.6.jar,file:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar</value>
</property>

<property>
<name>hive.metastore.uris</name>
<value>thrift://node17:9083</value>
</property>

<property>
<name>hive.exec.local.scratchdir</name>
<value>/home/hadoop/hive-1.2.1-bin/tmp</value>
</property>

</configuration>

A problem occurred:

Exception in thread "main"Java.lang.RuntimeException:java.lang.RuntimeException:Unable to instantiate Org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at Org.apache.hadoop.hive.ql.session.SessionState.start (Sessionstate.java:522) at Org.apache.hadoop.hive.cli.CliDriver.run (Clidriver.java:677) at Org.apache.hadoop.hive.cli.CliDriver.main (Clidriver.java:621) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke (Nativemethodaccessorimpl.java:57) at Sun.reflect.DelegatingMethodAccessorImpl.invoke (Delegatingmethodaccessorimpl.java:43) at Java.lang.reflect.Method.invoke (Method.java:30S) at Org.apache.hadoop.util.RunJar.run (Runjar.java:221) at Org.apache.hadoop.util.RunJar.main (Runjar.java:136) caused By:java.lang.RuntimeException:Unable to instantiate Org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Workaround:

Start the Metastore service.

./hive--service Metastore

Workaround:

Remove the Jline-0.9.94.jar jar package under yarn.

./hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar

Deploy hive on Apache hadoop2.6 and store the hive data source in MySQL