Manually install cloudera cdh4.2 hadoop + hbase + hive (3)

Source: Internet
Author: User
Tags psql xsl postgres database

This document describes how to manually install the cloudera hive cdh4.2.0 cluster. For environment setup and hadoop and hbase installation processes, see the previous article.

Install hive

Hive is installed on mongotop1. Note that hive saves metadata using the Derby database by default. Replace it with PostgreSQL here. The following describes how to install PostgreSQL, copy the Postgres jdbc jar file to the hive lib directory.

Upload files

Uploadhive-0.10.0-cdh4.2.0.tarTo strongtop1/opt, And decompress

Install ipvs
  • Create a database

Create the database MetaStore and hiveuser. The password is RedHat.

psql -U postgresCREATE DATABASE metastore; \c metastore;CREATE USER hiveuser WITH PASSWORD 'redhat';GRANT ALL ON DATABASE metastore TO hiveuser;\q
  • Initialize Database
psql  -U hiveuser -d metastore \i /opt/hive-0.10.0-cdh4.2.0/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql 
  • Edit the PostgreSQL configuration file and modify the access permission
[root@desktop1 ~]# vi /opt/PostgreSQL/9.1/data/pg_hba.conf# IPv4 local connections:host    all             all             0.0.0.0/0            md5[root@desktop1 ~]# vi postgresql.confstandard_conforming_strings = off
  • Restart ipvs
su -c '/opt/PostgreSQL/9.1/bin/pg_ctl -D /opt/PostgreSQL/9.1/data restart' postgres
  • Copy the s jdbc driver/opt/hive-0.10.0-cdh4.2.0/lib
Modify configuration file
  • Hive-site.xml pay attention to modify the password of the Postgres database in the configuration file below, pay attention to the configurationhive.aux.jars.pathTo integrate hbase in hive, You need to import some jar files in hbase from the path.
[root@desktop1 ~]# cd /opt/hive-0.10.0-cdh4.2.0/conf/[root@desktop1 conf]# cat hive-site.xml <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property>  <name>javax.jdo.option.ConnectionURL</name>  <value>jdbc:postgresql://127.0.0.1/metastore</value>  <description>JDBC connect string for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionDriverName</name>  <value>org.postgresql.Driver</value>  <description>Driver class name for a JDBC metastore</description></property><property>  <name>javax.jdo.option.ConnectionUserName</name>  <value>hiveuser</value>  <description>username to use against metastore database</description></property><property>  <name>javax.jdo.option.ConnectionPassword</name>  <value>redhat</value>  <description>password to use against metastore database</description></property><property> <name>mapred.job.tracker</name> <value>desktop1:8031</value></property><property> <name>mapreduce.framework.name</name> <value>yarn</value></property><property>  <name>hive.aux.jars.path</name>  <value>file:///opt/hive-0.10.0-cdh4.2.0/lib/zookeeper-3.4.5-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hbase-0.94.2-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/guava-11.0.2.jar</value></property><property>  <name>hive.metastore.warehouse.dir</name>  <value>/opt/data/warehouse-${user.name}</value>  <description>location of default database for the warehouse</description></property><property>  <name>hive.exec.scratchdir</name>  <value>/opt/data/hive-${user.name}</value>  <description>Scratch space for Hive jobs</description></property><property>  <name>hive.querylog.location</name>  <value>/opt/data/querylog-${user.name}</value>  <description>    Location of Hive run time structured log file  </description></property><property>  <name>hive.support.concurrency</name>  <description>Enable Hive's Table Lock Manager Service</description>  <value>true</value></property><property>  <name>hive.zookeeper.quorum</name>  <description>Zookeeper quorum used by Hive's Table Lock Manager</description>  <value>desktop3,desktop4,desktop6,desktop7,desktop8</value></property><property>  <name>hive.hwi.listen.host</name>  <value>desktop1</value>  <description>This is the host address the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.listen.port</name>  <value>9999</value>  <description>This is the port the Hive Web Interface will listen on</description></property><property>  <name>hive.hwi.war.file</name>  <value>lib/hive-hwi-0.10.0-cdh4.2.0.war</value>  <description>This is the WAR file with the jsp content for Hive Web Interface</description></property></configuration>
  • Environment Variable

Refer to setting environment variables in hadoop

  • Start script

After startup, some SQL statements may prompt errors. For how to solve the errors, see hive installation and configuration.

[root@desktop1 ~] hive
  • Hive and hbase are integrated inhive-site.xmlConfiguringhive.aux.jars.pathConfigure environment variables of hadoop and mapreduce in Environment Variables
Exception description
  • Failed: Error in metadata: metaexception (message: org. Apache. hadoop. hbase. zookeeperconnectionexception: an error is preventing hbase from connecting to zookeeper

Cause: the hadoop configuration file does not have ZK

  • Failed: Error in metadata: metaexception (message: Got exception: Org. apache. hadoop. hive. metaStore. API. metaexception javax. JDO. jdodatastoreexception: Error executing jdoql query "select" this ". "tbl_name" as nucorder0 from "Tsung" "this" left Outer Join "DBs"
    "This_database_name" on "this". "db_id" = "this_database_name". "db_id" where "this_database_name". "name" =? And (lower ("this". "tbl_name") like? Escape '\') order by nucorder0 ": Error: Invalid escape string suggestion: Escape string must be empty or one character ..

Reference: https://issues.apache.org/jira/browse/HIVE-3994

  • Hive> select count (*) from hive_userinfo; no response

  • Zookeeper. clientcnxn (clientcnxn. Java: logstartconnect (966)-opening socket connection to server localhost/127.0.0.1: 2181. will not attempt to authenticate using SASL (login configuration cannot be located)

Cause: ZK is not set in hive

  • Prompt in hbase: Warn util. nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable

Cause: cloudera hadoop lib does not contain hadoop native jar.

  • Exception in thread "Main" Java. Lang. noclassdeffounderror: ORG/Apache/hadoop/mapreduce/v2/APP/mrappmaster

Cause: the classpath is not correctly configured. Check the environment variables and the yarn classpath.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.