Hadoop cluster (CDH4) Practice (3) Hive Construction

Directory structure: Hadoop cluster (CDH4) practices (0) preface Hadoop cluster (CDH4) Practices (1) Hadoop (HDFS) Build Hadoop cluster (CDH4) practices (2) build Hadoop cluster (CDH4) using HBaseZookeeper (3) Build Hadoop cluster (CHD4) using Hive (4) Build Hadoop cluster (CHD4) using Oozie (5) Sqoop Security

Directory structure: Hadoop cluster (CDH4) practices (0) preface Hadoop cluster (CDH4) Practices (1) Hadoop (HDFS) Build Hadoop cluster (CDH4) practices (2) build Hadoop cluster (CDH4) using HBaseZookeeper (3) Build Hadoop cluster (CHD4) using Hive (4) Build Hadoop cluster (CHD4) using Oozie (5) Sqoop Security

Directory structure
Hadoop cluster (CDH4) practices (0) Preface
Hadoop cluster (CDH4) Practice (1) Hadoop (HDFS) Construction
Hadoop cluster (CDH4) Practice (2) HBase & Zookeeper Construction
Hadoop cluster (CDH4) Practice (3) Hive Construction
Hadoop cluster (CHD4) Practice (4) Oozie Construction
Hadoop cluster (CHD4) Practice (5) Sqoop Installation

Content
Hadoop cluster (CDH4) Practice (3) Hive Construction

References
Http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

Environment preparation
OS: CentOS 6.4 x86_64
Servers:
Hadoop-master: 172.17.0000230 memory 10 GB
-Namenode
-Hbase-master

Hadoop-secondary: 172.17.0000234 memory 10 GB
-Secondarybackupnamenode, jobtracker
-Hive-server, hive-metastore

Hadoop-node-1: 172.17.425231 memory 10G
-Datanode, tasktracker
-Hbase-regionserver, zookeeper-server

Hadoop-node-2: 172.17.20.memory 10 GB
-Datanode, tasktracker
-Hbase-regionserver, zookeeper-server

Hadoop-node-3: 172.17.425233 memory 10G
-Datanode, tasktracker
-Hbase-regionserver, zookeeper-server

Give a brief introduction to the above roles:
Namenode-namespace management service for the entire HDFS
Secondarynamenode-it can be seen as a namenode redundancy Service
Jobtracker-job Management Service for Parallel Computing
Datanode-HDFS node service
Tasktracker-job execution service for Parallel Computing
Hbase-master-Hbase Management Service
Hbase-regionServer-provides services such as Client insertion, deletion, and data query.
Zookeeper-server-Zookeeper collaboration and Configuration Management Service
Hive-server-Hive Management Service
Hive-metastore-Hive metadata, used for type check and syntax analysis of metadata

The specifications defined in this Article avoid confusion in understanding the configuration of multiple servers:
All of the following operations must be performed on the host where Hive is located, that is, hadoop-secondary.

1. Preparations before installation
Hadoop cluster (CDH4) Practice (2) HBase & Zookeeper Construction

2. Install Hive
$ Sudo yum install hive-metastore hive-server
$ Sudo yum install hive-jdbc hive-hbase

3. Install MySQL JDBC ctor
$ Sudo yum install mysql-connector-java
$ Sudo ln-s/usr/share/java/mysql-connector-java.jar/usr/lib/hive/lib/mysql-connector-java.jar

4. Install MySQL
$ Sudo yum install mysql-server
$ Sudo/etc/init. d/mysqld start

$ Sudo/usr/bin/mysql_secure_installation

 [...]Enter current password for root (enter for none):OK, successfully used password, moving on...[...]Set root password? [Y/n] yNew password: hiveserverRe-enter new password: hiverserverRemove anonymous users? [Y/n] Y[...]Disallow root login remotely? [Y/n] N[...]Remove test database and access to it [Y/n] Y[...]Reload privilege tables now? [Y/n] YAll done!

5. Create and authorize a database
$ Mysql-u root-phiveserver

 mysql> CREATE DATABASE metastore;mysql> USE metastore;mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%';mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost';mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver';mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1';mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';

6. Configure hive-site.xml
$ Sudo vim/etc/hive/conf/hive-site.xml

 
 
   javax.jdo.option.ConnectionURL  jdbc:mysql://hadoop-secondary/metastore  the URL of the MySQL database  javax.jdo.option.ConnectionDriverName  com.mysql.jdbc.Driver  javax.jdo.option.ConnectionUserName  hive  javax.jdo.option.ConnectionPassword  hiveserver  datanucleus.autoCreateSchema  false  datanucleus.fixedDatastore  true  datanucleus.autoStartMechanism   SchemaTable  hive.metastore.uris  thrift://hadoop-secondary:9083  IP address (or fully-qualified domain name) and port of the metastore host  hive.aux.jars.path  file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar  hbase.zookeeper.quorum  hadoop-node-1,hadoop-node-2,hadoop-node-3

7. Start Hive
$/Etc/init. d/hive-metastore start
$/Etc/init. d/hive-server start

8. Create the HDFS directory required by Hive
$ Sudo-u hdfs hadoop fs-mkdir/user/hive
$ Sudo-u hdfs hadoop fs-mkdir/user/hive/warehouse
$ Sudo-u hdfs hadoop fs-ls-R/user
$ Sudo-u hdfs hadoop fs-chown-R hive/user/hive
$ Sudo-u hdfs hadoop fs-chmod-R 1777/user/hive/warehouse

$ Sudo-u hdfs hadoop fs-chmod-R 777/tmp/hadoop-mapred
$ Sudo-u hdfs hadoop fs-chmod-R 777/tmp/hive-hive
$ Sudo chown-R hive: hive/var/lib/hive/. hivehistory

9. Now, Hive has been built.