Manually install clouderacdh4.2hadoop + hbase + hive (3)

This document describes how to manually install the clouderaHivecdh4.2.0 cluster. For environment setup and Hadoop and HBase installation processes, see the previous article. Install hivehive on desktop1. Note that hive saves metadata using the derby database by default. Replace it with postgresql here. The following describes how to install postgresql and copy postg.

This document describes how to manually install the cloudera Hive cdh4.2.0 cluster. For environment setup and Hadoop and HBase installation processes, see the previous article. Install hive on mongotop1. Note that hive saves metadata using the derby database by default. Replace it with postgresql here. The following describes how to install postgresql and copy postg.

This document describes how to manually install the cloudera Hive cdh4.2.0 cluster. For environment setup and Hadoop and HBase installation processes, see the previous article.

Install hive

Hive is installed on mongotop1. Note that hive saves metadata using the derby database by default. Replace it with postgresql here. The following describes how to install postgresql, copy the postgres jdbc jar file to the hive lib directory.

Upload files

Uploadhive-0.10.0-cdh4.2.0.tarTo strongtop1/opt, And decompress

Install ipvs

• Create a database

Create the database metastore and hiveuser. The password is redhat.

psql -U postgresCREATE DATABASE metastore; \c metastore;CREATE USER hiveuser WITH PASSWORD 'redhat';GRANT ALL ON DATABASE metastore TO hiveuser;\q

• Initialize Database

psql  -U hiveuser -d metastore \i /opt/hive-0.10.0-cdh4.2.0/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql 

• Edit the postgresql configuration file and modify the access permission

[root@desktop1 ~]# vi /opt/PostgreSQL/9.1/data/pg_hba.conf# IPv4 local connections:host    all             all             0.0.0.0/0            md5[root@desktop1 ~]# vi postgresql.confstandard_conforming_strings = off

• Restart ipvs

su -c '/opt/PostgreSQL/9.1/bin/pg_ctl -D /opt/PostgreSQL/9.1/data restart' postgres

• Copy the S jdbc driver/opt/hive-0.10.0-cdh4.2.0/lib

Modify configuration file

• Hive-site.xml pay attention to modify the password of the postgres database in the configuration file below, pay attention to the configurationhive.aux.jars.pathTo integrate hbase in hive, You need to import some jar files in hbase from the path.

[root@desktop1 ~]# cd /opt/hive-0.10.0-cdh4.2.0/conf/[root@desktop1 conf]# cat hive-site.xml 
  
  
  
     
    
     javax.jdo.option.ConnectionURL
      
    
     jdbc:postgresql://127.0.0.1/metastore
      
    
     JDBC connect string for a JDBC metastore
    
   
     
    
     javax.jdo.option.ConnectionDriverName
      
    
     org.postgresql.Driver
      
    
     Driver class name for a JDBC metastore
    
   
     
    
     javax.jdo.option.ConnectionUserName
      
    
     hiveuser
      
    
     username to use against metastore database
    
   
     
    
     javax.jdo.option.ConnectionPassword
      
    
     redhat
      
    
     password to use against metastore database
    
   
    
    
     mapred.job.tracker
     
    
     desktop1:8031
    
   
    
    
     mapreduce.framework.name
     
    
     yarn
    
   
     
    
     hive.aux.jars.path
      
    
     file:///opt/hive-0.10.0-cdh4.2.0/lib/zookeeper-3.4.5-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hive-hbase-handler-0.10.0-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/hbase-0.94.2-cdh4.2.0.jar,    file:///opt/hive-0.10.0-cdh4.2.0/lib/guava-11.0.2.jar
    
   
     
    
     hive.metastore.warehouse.dir
      
    
     /opt/data/warehouse-${user.name}
      
    
     location of default database for the warehouse
    
   
     
    
     hive.exec.scratchdir
      
    
     /opt/data/hive-${user.name}
      
    
     Scratch space for Hive jobs
    
   
     
    
     hive.querylog.location
      
    
     /opt/data/querylog-${user.name}
      
    
         Location of Hive run time structured log file  
    
   
     
    
     hive.support.concurrency
      
    
     Enable Hive's Table Lock Manager Service
      
    
     true
    
   
     
    
     hive.zookeeper.quorum
      
    
     Zookeeper quorum used by Hive's Table Lock Manager
      
    
     desktop3,desktop4,desktop6,desktop7,desktop8
    
   
     
    
     hive.hwi.listen.host
      
    
     desktop1
      
    
     This is the host address the Hive Web Interface will listen on
    
   
     
    
     hive.hwi.listen.port
      
    
     9999
      
    
     This is the port the Hive Web Interface will listen on
    
   
     
    
     hive.hwi.war.file
      
    
     lib/hive-hwi-0.10.0-cdh4.2.0.war
      
    
     This is the WAR file with the jsp content for Hive Web Interface
    
   
  

• Environment Variable

Refer to setting environment variables in hadoop

• Start script

After startup, some SQL statements may prompt errors. For how to solve the errors, see Hive installation and configuration.

[root@desktop1 ~] hive

• Hive and hbase are integrated inhive-site.xmlConfiguringhive.aux.jars.pathConfigure environment variables of hadoop and mapreduce in Environment Variables

Exception description

• FAILED: Error in metadata: MetaException (message: org. apache. hadoop. hbase. ZooKeeperConnectionException: An error is preventing HBase from connecting to ZooKeeper

Cause: the hadoop configuration file does not have zk

• FAILED: Error in metadata: MetaException (message: Got exception: org. apache. hadoop. hive. metastore. api. metaException javax. jdo. JDODataStoreException: Error executing JDOQL query "SELECT" THIS ". "TBL_NAME" AS NUCORDER0 FROM "Tsung" "THIS" left outer join "DBS" "THIS_DATABASE_NAME" ON "THIS ". "DB_ID" = "THIS_DATABASE_NAME ". "DB_ID" WHERE "THIS_DATABASE_NAME ". "NAME" =? AND (LOWER ("THIS". "TBL_NAME") LIKE? ESCAPE '\') order by NUCORDER0 ": ERROR: invalid escape string suggestion: Escape string must be empty or one character ..

Reference: https://issues.apache.org/jira/browse/HIVE-3994

• Hive> select count (*) from hive_userinfo; no response

• Zookeeper. ClientCnxn (ClientCnxn. java: logStartConnect (966)-Opening socket connection to server localhost/127.0.0.1: 2181. Will not attempt to authenticate using SASL (login configuration cannot be located)

Cause: zk is not set in hive

• Prompt in hbase: WARN util. NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Cause: cloudera hadoop lib does not contain hadoop native jar.

• Exception in thread "main" java. lang. NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster

Cause: the classpath is not correctly configured. Check the environment variables and the yarn classpath.