手動安裝cloudera cdh4.2 hadoop + hbase + hive（三）

本文主要記錄手動安裝cloudera Hive cdh4.2.0叢集過程，環境設定及Hadoop、HBase安裝過程見上篇文章。

安裝hive

hive安裝在desktop1上，注意hive預設是使用derby資料庫儲存中繼資料，這裡替換為postgresql，下面會提到postgresql的安裝說明，並且需要拷貝postgres的jdbc jar檔案導hive的lib目錄下。

上傳檔案

上傳hive-0.10.0-cdh4.2.0.tar到desktop1的/opt，並解壓縮

安裝postgres

• 建立資料庫

這裡建立資料庫metastore並建立hiveuser使用者，其密碼為redhat。

psql -U postgresCREATE DATABASE metastore; \c metastore;CREATE USER hiveuser WITH PASSWORD 'redhat';GRANT ALL ON DATABASE metastore TO hiveuser;\q

• 初始化資料庫

psql  -U hiveuser -d metastore \i /opt/hive-0.10.0-cdh4.2.0/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql 

• 編輯postgresql設定檔，修改存取權限

[root@desktop1 ~]# vi /opt/PostgreSQL/9.1/data/pg_hba.conf# IPv4 local connections:host    all             all             0.0.0.0/0            md5[root@desktop1 ~]# vi postgresql.confstandard_conforming_strings = off

• 重起postgres

su -c '/opt/PostgreSQL/9.1/bin/pg_ctl -D /opt/PostgreSQL/9.1/data restart' postgres

• 拷貝postgres 的jdbc驅動到/opt/hive-0.10.0-cdh4.2.0/lib

修改設定檔

• hive-site.xml 注意修改下面設定檔中postgres資料庫的密碼，注意配置hive.aux.jars.path，在hive整合hbase時候需要從該路徑家在hbase的一些jar檔案。

[root@desktop1 ~]# cd /opt/hive-0.10.0-cdh4.2.0/conf/[root@desktop1 conf]# cat hive-site.xml <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property>  <name>javax.jdo.option.ConnectionURL</name>  <value>jdbc:postgresql://127.0.0.1/metastore</value>  <description>JDBC connect string for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionDriverName</name>  <value>org.postgresql.Driver</value>  <description>Driver class name for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionUserName</name>  <value>hiveuser</value>  <description>username to use against metastore database</description></property><property>  <name>javax.jdo.option.ConnectionPassword</name>  <value>redhat</value>  <description>password to use against metastore database</description></property><property> <name>mapred.job.tracker</name> <value>desktop1:8031</value></property><property> <name>mapreduce.framework.name</name> <value>yarn</value></property><property>  <name>hive.aux.jars.path</name>  <value>file:///opt/hive-0.10.0-cdh4.2.0/lib/zookeeper-3.4.5-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hbase-0.94.2-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/guava-11.0.2.jar</value></property><property>  <name>hive.metastore.warehouse.dir</name>  <value>/opt/data/warehouse-${user.name}</value>  <description>location of default database for the warehouse</description></property><property>  <name>hive.exec.scratchdir</name>  <value>/opt/data/hive-${user.name}</value>  <description>Scratch space for Hive jobs</description></property><property>  <name>hive.querylog.location</name>  <value>/opt/data/querylog-${user.name}</value>  <description>    Location of Hive run time structured log file  </description></property><property>  <name>hive.support.concurrency</name>  <description>Enable Hive's Table Lock Manager Service</description>  <value>true</value></property><property>  <name>hive.zookeeper.quorum</name>  <description>Zookeeper quorum used by Hive's Table Lock Manager</description>  <value>desktop3,desktop4,desktop6,desktop7,desktop8</value></property><property>  <name>hive.hwi.listen.host</name>  <value>desktop1</value>  <description>This is the host address the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.listen.port</name>  <value>9999</value>  <description>This is the port the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.war.file</name>  <value>lib/hive-hwi-0.10.0-cdh4.2.0.war</value>  <description>This is the WAR file with the jsp content for Hive Web Interface</description></property></configuration>

• 環境變數

參考hadoop中環境變數的設定

• 啟動指令碼

在啟動完之後，執行一些sql語句可能會提示錯誤，如何解決錯誤可以參考Hive安裝與配置。

[root@desktop1 ~] hive

• hive與hbase整合 在hive-site.xml中配置hive.aux.jars.path,在環境變數中配置hadoop、mapreduce的環境變數

異常說明

• FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is preventing HBase from connecting to ZooKeeper

原因：hadoop設定檔沒有zk

• FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDODataStoreException: Error executing JDOQL query "SELECT "THIS"."TBL_NAME" AS NUCORDER0 FROM "TBLS" "THIS" LEFT OUTER JOIN "DBS"
  "THIS_DATABASE_NAME" ON "THIS"."DB_ID" = "THIS_DATABASE_NAME"."DB_ID" WHERE "THIS_DATABASE_NAME"."NAME" = ? AND (LOWER("THIS"."TBL_NAME") LIKE ? ESCAPE '\' ) ORDER BY NUCORDER0 " : ERROR: invalid escape string 建議：Escape string must be empty or one character..

參考：https://issues.apache.org/jira/browse/HIVE-3994

• hive> select count(*) from hive_userinfo; 沒反應

• zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(966)) - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (無法定位登入配置)

原因：hive中沒有設定zk

• hbase 中提示：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

原因：cloudera hadoop lib中沒有hadoop的native jar

• Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster

原因：classpath沒有配置正確，檢查環境變數以及yarn的classpath