Manually install cloudera cdh4.2 hadoop + hbase + hive (3)

This document describes how to manually install the cloudera hive cdh4.2.0 cluster. For environment setup and hadoop and hbase installation processes, see the previous article.

Install hive

Hive is installed on mongotop1. Note that hive saves metadata using the Derby database by default. Replace it with PostgreSQL here. The following describes how to install PostgreSQL, copy the Postgres jdbc jar file to the hive lib directory.

Upload files

Uploadhive-0.10.0-cdh4.2.0.tarTo strongtop1/opt, And decompress

Install ipvs

• Create a database

Create the database MetaStore and hiveuser. The password is RedHat.

psql -U postgresCREATE DATABASE metastore; \c metastore;CREATE USER hiveuser WITH PASSWORD 'redhat';GRANT ALL ON DATABASE metastore TO hiveuser;\q

• Initialize Database

psql  -U hiveuser -d metastore \i /opt/hive-0.10.0-cdh4.2.0/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql 

• Edit the PostgreSQL configuration file and modify the access permission

[root@desktop1 ~]# vi /opt/PostgreSQL/9.1/data/pg_hba.conf# IPv4 local connections:host    all             all             0.0.0.0/0            md5[root@desktop1 ~]# vi postgresql.confstandard_conforming_strings = off

• Restart ipvs

su -c '/opt/PostgreSQL/9.1/bin/pg_ctl -D /opt/PostgreSQL/9.1/data restart' postgres

• Copy the s jdbc driver/opt/hive-0.10.0-cdh4.2.0/lib

Modify configuration file

• Hive-site.xml pay attention to modify the password of the Postgres database in the configuration file below, pay attention to the configurationhive.aux.jars.pathTo integrate hbase in hive, You need to import some jar files in hbase from the path.

[root@desktop1 ~]# cd /opt/hive-0.10.0-cdh4.2.0/conf/[root@desktop1 conf]# cat hive-site.xml <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property>  <name>javax.jdo.option.ConnectionURL</name>  <value>jdbc:postgresql://127.0.0.1/metastore</value>  <description>JDBC connect string for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionDriverName</name>  <value>org.postgresql.Driver</value>  <description>Driver class name for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionUserName</name>  <value>hiveuser</value>  <description>username to use against metastore database</description></property><property>  <name>javax.jdo.option.ConnectionPassword</name>  <value>redhat</value>  <description>password to use against metastore database</description></property><property> <name>mapred.job.tracker</name> <value>desktop1:8031</value></property><property> <name>mapreduce.framework.name</name> <value>yarn</value></property><property>  <name>hive.aux.jars.path</name>  <value>file:///opt/hive-0.10.0-cdh4.2.0/lib/zookeeper-3.4.5-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hbase-0.94.2-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/guava-11.0.2.jar</value></property><property>  <name>hive.metastore.warehouse.dir</name>  <value>/opt/data/warehouse-${user.name}</value>  <description>location of default database for the warehouse</description></property><property>  <name>hive.exec.scratchdir</name>  <value>/opt/data/hive-${user.name}</value>  <description>Scratch space for Hive jobs</description></property><property>  <name>hive.querylog.location</name>  <value>/opt/data/querylog-${user.name}</value>  <description>    Location of Hive run time structured log file  </description></property><property>  <name>hive.support.concurrency</name>  <description>Enable Hive's Table Lock Manager Service</description>  <value>true</value></property><property>  <name>hive.zookeeper.quorum</name>  <description>Zookeeper quorum used by Hive's Table Lock Manager</description>  <value>desktop3,desktop4,desktop6,desktop7,desktop8</value></property><property>  <name>hive.hwi.listen.host</name>  <value>desktop1</value>  <description>This is the host address the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.listen.port</name>  <value>9999</value>  <description>This is the port the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.war.file</name>  <value>lib/hive-hwi-0.10.0-cdh4.2.0.war</value>  <description>This is the WAR file with the jsp content for Hive Web Interface</description></property></configuration>

• Environment Variable

Refer to setting environment variables in hadoop

• Start script

After startup, some SQL statements may prompt errors. For how to solve the errors, see hive installation and configuration.

[root@desktop1 ~] hive

• Hive and hbase are integrated inhive-site.xmlConfiguringhive.aux.jars.pathConfigure environment variables of hadoop and mapreduce in Environment Variables

Exception description

• Failed: Error in metadata: metaexception (message: org. Apache. hadoop. hbase. zookeeperconnectionexception: an error is preventing hbase from connecting to zookeeper

Cause: the hadoop configuration file does not have ZK

• Failed: Error in metadata: metaexception (message: Got exception: Org. apache. hadoop. hive. metaStore. API. metaexception javax. JDO. jdodatastoreexception: Error executing jdoql query "select" this ". "tbl_name" as nucorder0 from "Tsung" "this" left Outer Join "DBs"
  "This_database_name" on "this". "db_id" = "this_database_name". "db_id" where "this_database_name". "name" =? And (lower ("this". "tbl_name") like? Escape '\') order by nucorder0 ": Error: Invalid escape string suggestion: Escape string must be empty or one character ..

Reference: https://issues.apache.org/jira/browse/HIVE-3994

• Hive> select count (*) from hive_userinfo; no response

• Zookeeper. clientcnxn (clientcnxn. Java: logstartconnect (966)-opening socket connection to server localhost/127.0.0.1: 2181. will not attempt to authenticate using SASL (login configuration cannot be located)

Cause: ZK is not set in hive

• Prompt in hbase: Warn util. nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable

Cause: cloudera hadoop lib does not contain hadoop native jar.

• Exception in thread "Main" Java. Lang. noclassdeffounderror: ORG/Apache/hadoop/mapreduce/v2/APP/mrappmaster

Cause: the classpath is not correctly configured. Check the environment variables and the yarn classpath.