[Hive]那些年我們踩過的Hive坑

1. 缺少MySQL驅動包 1.1 問題描述

Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)at org.datanucleus.store.rdbms.connectionpool.BoneCPConnectionPoolFactory.createConnectionPool(BoneCPConnectionPoolFactory.java:54)at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)

1.2. 解決方案

上述問題很可能是缺少mysql的jar包，下載mysql-connector-java-5.1.32.tar.gz，複製到hive的lib目錄下：

xiaosi@yoona:~$ cp mysql-connector-java-5.1.34-bin.jar opt/hive-2.1.0/lib/

2. 中繼資料庫mysql初始化 2.1 問題描述

運行./hive指令碼時，無法進入，報錯：

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

2.2 解決方案

在scripts目錄下運行 schematool -initSchema -dbType mysql命令進行Hive中繼資料庫的初始化：

xiaosi@yoona:~/opt/hive-2.1.0/scripts$  schematool -initSchema -dbType mysqlSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/xiaosi/opt/hive-2.1.0/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/xiaosi/opt/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Metastore connection URL: jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=trueMetastore Connection Driver : com.mysql.jdbc.DriverMetastore connection User: rootStarting metastore schema initialization to 2.1.0Initialization script hive-schema-2.1.0.mysql.sqlInitialization script completedschemaTool completed

3. Relative path in absolute URI 3.1 問題描述

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D...Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7Dat java.net.URI.checkPath(URI.java:1823)at java.net.URI.<init>(URI.java:745)at org.apache.hadoop.fs.Path.initialize(Path.java:202)... 12 more

3.2 解決方案

產生上述問題的原因是使用了沒有配置的變數，解決此問題只需在設定檔hive-site.xml中配置system:user.name和system:java.io.tmpdir兩個變數，設定檔中就可以使用這兩個變數：

<property>    <name>system:user.name</name>    <value>xiaosi</value></property><property>    <name>system:java.io.tmpdir</name>    <value>/home/${system:user.name}/tmp/hive/</value></property>

4. 拒絕串連 4.1 問題描述

on exception: java.net.ConnectException: 拒絕串連; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused...Caused by: java.net.ConnectException: Call From Qunar/127.0.0.1 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒絕串連; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused...Caused by: java.net.ConnectException: 拒絕串連at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)at org.apache.hadoop.net.