Hadoop (10)-Hive installation vs. custom functions

A. Hive installation

Hive is installed on only one node

1. Upload the TAR package
2. Unzip TAR-ZXVF hive-0.9.0.tar.gz-c/cloud/
3. Configure MySQL metastore (switch to root user)

3.1 Configuring Hive_home Environment variables

3.2 Installing MySQL

querying previously installed MySQL-related packages : Rpm-qa | grep MySQL
violence to remove this package : Rpm-e mysql-libs-5.1.66-2.el6_3.i686--nodeps
Installing MYSQL:RPM-IVH mysql-server-5.1.73-1.glibc23.i386.rpm
RPM-IVH mysql-client-5.1.73-1.glibc23.i386.rpm
Execute command settings mysql:/usr/bin/mysql_secure_installation (Note: Remove anonymous users, allow users to connect remotely)
Login Mysql:mysql-u Root-p

4. Configure Hive
CP Hive-default.xml.template Hive-site.xml
Modify Hive-site.xml (delete all content, leave only one <property></property>)

Add the following content:

<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://hadoop00:3306 /hive?createdatabaseifnotexist=true</value> &LT;DESCRIPTION&GT;JDBC connect string for a JDBC metastore</  Description></property><property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>driver class name for a JDBC metastore</  Description></property><property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against Metastore database</description> </property><property> <name>javax.jdo.option.ConnectionPassword</name> <value>123 </value> <description>password to use against Metastore database</description></property>

5. After installing HIVE and mysq, copy the MySQL connection jar package to the $hive_home/lib directory
If there is an issue with no permissions, in MySQL authorization (performed on the machine where MySQL is installed)
Mysql-uroot-p
# (Execute the following statement * *: All tables under all libraries%: Any IP address or host can be connected)
GRANT all privileges on * * to ' root ' @ '% ' identified by ' 123 ' with GRANT OPTION;
FLUSH privileges;

6. Build table (default is internal table)
CREATE TABLE Trade_detail (ID bigint, account string, income double, expenses double, time string) row format delimited fie lds terminated by ' \ t ';

Build partition Table:

CREATE TABLE Td_part (ID bigint, account string, income double, expenses double, time string) partitioned by (Logdate Strin g) Row format delimited fields terminated by ' \ t ';

Build External table:

Create external Table td_ext (ID bigint, account string, income double, expenses double, time string) row format Delimi Ted fields terminated by ' \ t ' location '/td_ext ';

7. Create a partitioned table
Common table and partition table differences: There is a large amount of data needed to build a partitioned table
CREATE TABLE book (id bigint, name string) partitioned by (pubdate string) row format delimited fields terminated by ' \ t ';

Partition Table Load Data:
Load data local inpath './book.txt ' overwrite to table book partition (pubdate= ' 2010-08-22 ');
Load data local inpath '/root/data.am ' into table beauty partition (nation= "USA");

Select Nation, AVG (size) from the Beauties group by Nation ORDER by AVG (size);

Two. UDF

Custom UDF to inherit the Org.apache.hadoop.hive.ql.exec.UDF class implementation evaluate

public class Areaudf extends Udf{private static Map<integer, string> Areamap = new Hashmap<integer, string> (); static {Areamap.put (1, "Beijing"), Areamap.put (2, "Shanghai"), Areamap.put (3, "Guangzhou");} Public Text evaluate (text in) {String result = Areamap.get (Integer.parseint (in.tostring ())); if (result = = null) {result = " Other ";} return new Text (result);}}

Custom Function Call Procedure:
1. Add a jar package (executed in the Hive command line)
hive> add Jar/root/nudf.jar;

2. Create a temporary function
Hive> Create temporary function getnation as ' Cn.itcast.hive.udf.NationUDF ';

3. Call
Hive> Select ID, Name, getnation (Nation) from beauty;

4. Save the query results in HDFs
hive> CREATE table result row format delimited fields terminated by ' \ t ' as SELECT * from beauty ORDER by id desc;
Hive> Select ID, getareaname (ID) as name from Tel_rec;
Create table result row format delimited fields terminated by ' \ t ' as select ID, getnation (Nation) from beauties;

Hadoop (10)-Hive installation vs. custom functions