This document describes how to manually install the cloudera hive cdh4.2.0 cluster. For environment setup and hadoop and hbase installation processes, see the previous article.
Install hive
Hive is installed on mongotop1. Note that hive saves metadata using the Derby database by default. Replace it with PostgreSQL here. The following describes how to install PostgreSQL, copy the Postgres jdbc jar file to the hive lib directory.
Upload files
Uploadhive-0.10.0-cdh4.2.0.tar
To strongtop1/opt
, And decompress
Install ipvs
Create the database MetaStore and hiveuser. The password is RedHat.
psql -U postgresCREATE DATABASE metastore; \c metastore;CREATE USER hiveuser WITH PASSWORD 'redhat';GRANT ALL ON DATABASE metastore TO hiveuser;\q
psql -U hiveuser -d metastore \i /opt/hive-0.10.0-cdh4.2.0/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql
- Edit the PostgreSQL configuration file and modify the access permission
[root@desktop1 ~]# vi /opt/PostgreSQL/9.1/data/pg_hba.conf# IPv4 local connections:host all all 0.0.0.0/0 md5[root@desktop1 ~]# vi postgresql.confstandard_conforming_strings = off
su -c '/opt/PostgreSQL/9.1/bin/pg_ctl -D /opt/PostgreSQL/9.1/data restart' postgres
- Copy the s jdbc driver
/opt/hive-0.10.0-cdh4.2.0/lib
Modify configuration file
- Hive-site.xml pay attention to modify the password of the Postgres database in the configuration file below, pay attention to the configuration
hive.aux.jars.path
To integrate hbase in hive, You need to import some jar files in hbase from the path.
[root@desktop1 ~]# cd /opt/hive-0.10.0-cdh4.2.0/conf/[root@desktop1 conf]# cat hive-site.xml <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://127.0.0.1/metastore</value> <description>JDBC connect string for a JDBC metastore</description></property><property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> <description>Driver class name for a JDBC metastore</description></property><property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value> <description>username to use against metastore database</description></property><property> <name>javax.jdo.option.ConnectionPassword</name> <value>redhat</value> <description>password to use against metastore database</description></property><property> <name>mapred.job.tracker</name> <value>desktop1:8031</value></property><property> <name>mapreduce.framework.name</name> <value>yarn</value></property><property> <name>hive.aux.jars.path</name> <value>file:///opt/hive-0.10.0-cdh4.2.0/lib/zookeeper-3.4.5-cdh4.2.0.jar, file:///opt/hive-0.10.0-cdh4.2.0/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar, file:///opt/hive-0.10.0-cdh4.2.0/lib/hbase-0.94.2-cdh4.2.0.jar, file:///opt/hive-0.10.0-cdh4.2.0/lib/guava-11.0.2.jar</value></property><property> <name>hive.metastore.warehouse.dir</name> <value>/opt/data/warehouse-${user.name}</value> <description>location of default database for the warehouse</description></property><property> <name>hive.exec.scratchdir</name> <value>/opt/data/hive-${user.name}</value> <description>Scratch space for Hive jobs</description></property><property> <name>hive.querylog.location</name> <value>/opt/data/querylog-${user.name}</value> <description> Location of Hive run time structured log file </description></property><property> <name>hive.support.concurrency</name> <description>Enable Hive's Table Lock Manager Service</description> <value>true</value></property><property> <name>hive.zookeeper.quorum</name> <description>Zookeeper quorum used by Hive's Table Lock Manager</description> <value>desktop3,desktop4,desktop6,desktop7,desktop8</value></property><property> <name>hive.hwi.listen.host</name> <value>desktop1</value> <description>This is the host address the Hive Web Interface will listen on</description></property><property> <name>hive.hwi.listen.port</name> <value>9999</value> <description>This is the port the Hive Web Interface will listen on</description></property><property> <name>hive.hwi.war.file</name> <value>lib/hive-hwi-0.10.0-cdh4.2.0.war</value> <description>This is the WAR file with the jsp content for Hive Web Interface</description></property></configuration>
Refer to setting environment variables in hadoop
After startup, some SQL statements may prompt errors. For how to solve the errors, see hive installation and configuration.
[root@desktop1 ~] hive
- Hive and hbase are integrated in
hive-site.xml
Configuringhive.aux.jars.path
Configure environment variables of hadoop and mapreduce in Environment Variables
Exception description
- Failed: Error in metadata: metaexception (message: org. Apache. hadoop. hbase. zookeeperconnectionexception: an error is preventing hbase from connecting to zookeeper
Cause: the hadoop configuration file does not have ZK
- Failed: Error in metadata: metaexception (message: Got exception: Org. apache. hadoop. hive. metaStore. API. metaexception javax. JDO. jdodatastoreexception: Error executing jdoql query "select" this ". "tbl_name" as nucorder0 from "Tsung" "this" left Outer Join "DBs"
"This_database_name" on "this". "db_id" = "this_database_name". "db_id" where "this_database_name". "name" =? And (lower ("this". "tbl_name") like? Escape '\') order by nucorder0 ": Error: Invalid escape string suggestion: Escape string must be empty or one character ..
Reference: https://issues.apache.org/jira/browse/HIVE-3994
Hive> select count (*) from hive_userinfo; no response
Zookeeper. clientcnxn (clientcnxn. Java: logstartconnect (966)-opening socket connection to server localhost/127.0.0.1: 2181. will not attempt to authenticate using SASL (login configuration cannot be located)
Cause: ZK is not set in hive
- Prompt in hbase: Warn util. nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable
Cause: cloudera hadoop lib does not contain hadoop native jar.
- Exception in thread "Main" Java. Lang. noclassdeffounderror: ORG/Apache/hadoop/mapreduce/v2/APP/mrappmaster
Cause: the classpath is not correctly configured. Check the environment variables and the yarn classpath.