(i) Installation, deployment and testing of hive

1. Installation EnvironmentJDK1.7.0 centOS6.4 hive0.13.1 cdh5.3.6 hadoop2.5.0 MySQL 2, Hive functional Framework Introduction

Hive is a tool for any size data analysis in SQL style, characterized by the use of SQL commands similar to relational databases. It is characterized by the large data of Hadoop processed through SQL, the scale of data can be scaled to 100pb+, the data form can be structural or unstructured data.

Hive and traditional relational database have the following characteristics:

> Focus on analysis, not real-time online transactions without transaction mechanism;
> Unlike relational databases, you can randomly insert or update;
> Distributed processing via Hadoop map/reduce, traditional database;
> Traditional relational databases can only expand up to 20 servers, while hive extends to hundreds of servers.

Comparison between Hive and MapReduce

Hive only need to write a few SQL statements to achieve the use of Java development mapreduce function, simplified the amount of code, reduce the development efficiency, for unfamiliar Java and the development of MapReduce has great benefits.

The hive's architectural drawings are as follows:
3, Hive installation Deployment 3.1. Configure the configuration file directory associated with Hadoop and hive

hive_home/conf/hive-env.sh file

# Set Hadoop_home to point to a specific HADOOP install directory
hadoop_home=/home/xusy/share/cdh5.3.6/ hadoop-2.5.0-cdh5.3.6

# Hive Configuration Directory can be controlled by:
Export hive_conf_dir=/home/xusy/ Share/cdh5.3.6/hive-0.13.1-cdh5.3.6/conf

3.2. Start the Hadoop process

$ sbin/hadoop-daemon.sh Start Namenode
$ sbin/hadoop-daemon.sh start Datanode
$ sbin/yarn-daemon.sh Start ResourceManager
$ sbin/yarn-daemon.sh start NodeManager
$ sbin/mr-jobhistory-daemon.sh Start historyserver

3.3, before creating the necessary operation of the table

$bin/hadoop fs-mkdir/       tmp
$bin/hadoop fs-mkdir       /user/hive/warehouse
$bin/hadoop fs-chmod g+w /   tmp
$bin/hadoop fs-chmod g+w   /user/hive/warehouse

4, DML/DDL statement test 4.1. Create a table

Hive>drop table if exists student;
Hive>create TABLE Student (
hive>id int,
hive>name string)
hive>row FORMAT delimited 
hive >fields terminated by ', ';

4.2. Prepare data

Stu.txt

01,xusy
02,liyj
03,liujl
04,hunl
05,zhaoyy
06,zhoujq 07,yuang 08,zhangz

Through the hue upload to HDFs/user/xusy/data, if not installed hue, then use the command to upload;
The path where the data source is stored in Hdgs is as follows:
4.3. Loading Data

Hive>load data inpath '/user/xusy/data/stu.txt ' overwrite into table student;

4.4, Inquiry

Hive>select count (1) from student;

5. Install MySQL under Linux 5.1, yum installation

$sudo yum-y Install Mysql-server

5.2, temporary start mysqld

Switch to root user
# service mysqld start

5.3, set up boot mysqld

# chkconfig mysqld on
# chkconfig--list|grep mysqld

5.4. Create MySQL user and password

$mysqladmin-uroot Password root

5.5. Start MySQL

$mysql-uroot-proot

5.6, modify the MySQL password (optional)

mysql> SET PASSWORD = PASSWORD (' 123456 ');

5.7, modify the MySQL database under the user table stored information 5.7.1, query user information

Mysql>use MySQL;
Mysql> select Password,host,user from User;

5.7.2, modify user information, modify to share a password for task users

mysql> Update user set host= '% ' where user= ' root ' and host= ' localhost ';

5.7.3, delete user information 1

mysql> Delete from user where user= ' root ' and host= ' 127.0.0.1 ';
Mysql> select Password,host,user from User;

5.7.4, delete user Information 2

mysql> Delete from user where user= ' root ' and host= ' xuxudede.com '; 
Mysql> select Password,host,user from User;

5.7.5, delete user Information 3

mysql> Delete from user where host= ' xuxudede.com '; 
Mysql> select Password,host,user from User;

5.7.6, delete user Information 4

mysql> Delete from user where host= ' localhost '; 
Mysql> select Password,host,user from User;

5.7.7, refreshing

Mysql>flush privileges;

6, configure the hive metadata stored in MySQL 6.1, hive-site.xml configuration file

Note: If the hive-site.xml is not present under hive_home/conf, create it yourself.
The configuration is as follows:

<property> <name>javax.jdo.option.ConnectionURL</name> <value> jdbc:mysql://xuxudede.com/metadata?createdatabaseifnotexist=true</value> <description>JDBC Connect String for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.conne Ctiondrivername</name> <value>com.mysql.jdbc.Driver</value> <description>driver class name For a JDBC metastore</description> </property> <property> <name> Javax.jdo.option.connectionusername</name> <value>root</value> <description>username to Use against Metastore database</description> </property> <property> <name> Javax.jdo.option.connectionpassword</name> <value>123456</value> <description>password to Use against Metastore database</description> </property> 

6.2, add MySQL driver 6.2.1, unzip the MySQL driver package, copy the jar package in MySQL to the Lib directory of the hive root directory

$tar-ZXVF mysql-connector-java-5.1.27.tar.gz
$ cp mysql-connector-java-5.1.27/ Mysql-connector-java-5.1.27-bin.jar.. /.. /cdh5.3.6/hive-0.13.1-cdh5.3.6/lib/

7, restart the hive, and the corresponding verification test 7.1. Reboot Hive

$bin/hive

7.2, view the data stored in MySQL metadata information

mysql> use metadata;

Mysql> select * from TBLs;
mysql> use TBLs;

At this point, the hive has been successfully installed and the MySQL storage hive metadata has been configured.