Alex's Hadoop rookie Tutorial: 8th Sqoop1 import Hbase and Hive

Continue writing. In fact, importing and exporting hdfs from mysql is useless for actual project development, but that can be used for getting started. Today we are writing about collaboration with Hbase and Hive. I suddenly found that the sequence of my tutorials was messy. I didn't introduce the installation of Hive first. I am sorry for this. I will make it up later. Data Preparation mysql creates the table "employee" in mysql.

Continue writing. In fact, importing and exporting hdfs from mysql is useless for actual project development, but that can be used for getting started. Today we are writing about collaboration with Hbase and Hive. I suddenly found that the sequence of my tutorials was messy. I didn't introduce the installation of Hive first. I am sorry for this. I will make it up later. Data Preparation mysql creates the table "employee" in mysql.

Continue writing. In fact, importing and exporting hdfs from mysql is useless for actual project development, but that can be used for getting started. Today we are writing about collaboration with Hbase and Hive. I suddenly found that the sequence of my tutorials was messy. I didn't introduce the installation of Hive first. I am sorry for this. I will make it up later.

Data Preparation mysql creates a table "employee" in mysql and inserts data.

CREATE TABLE `employee` (      `id` int(11) NOT NULL,      `name` varchar(20) NOT NULL,      PRIMARY KEY (`id`)    ) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  

insert into employee (id,name) values (1,'michael');  insert into employee (id,name) values (2,'ted'); insert into employee (id,name) values (3,'jack'); 

Hbase

hbase(main):006:0> create 'employee','info'0 row(s) in 0.4440 seconds=> Hbase::Table - employee

Hive does not need data preparation, etc. -- create-hive-table will automatically create the table

Import data from mysql to Hbase

# Sqoop import -- connect jdbc: mysql: // localhost: 3306/sqoop_test -- username root -- password root -- table employee -- hbase-table employee -- column-family info -- hbase-row-key id-m 1 Warning: /usr/lib/sqoop /.. /hive-hcatalog does not exist! HCatalog jobs will fail. Please set $ HCAT_HOME to the root of your HCatalog installation. Warning:/usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. please set $ ACCUMULO_HOME to the root of your Accumulo installation.14/12/01 17:36:25 INFO sqoop. sqoop: Running Sqoop version: 1.4.4-cdh5.0.114/12/01 17:36:25 WARN tool. baseSqoopTool: Setting your password on the command-line is insecure. consider using-P instead.14/12/01 17:36:25 INFO manager. mySQLManager: Preparing to use a MySQL streaming resultset.14/12/01 17:36:25 INFO tool. codeGenTool: Beginning code generation14/12/01 17:36:26 INFO manager. sqlManager: Executing SQL statement: SELECT t. * FROM 'Employee' AS t LIMIT 114/12/01 17:36:26 INFO manager. sqlManager: Executing SQL statement: SELECT t. * FROM 'Employee' AS t LIMIT 114/12/01 17:36:26 INFO orm. compilationManager: HADOOP_MAPRED_HOME is/usr/lib/hadoop-mapreduce ...... Too many intermediate logs, replacing 14/12/01 17:37:12 INFO mapreduce with ellipsis. importJobBase: Transferred 0 bytes in 37.3924 seconds (0 bytes/sec) 14/12/01 17:37:12 INFO mapreduce. importJobBase: Retrieved 3 records.

Check hbase

hbase(main):001:0> scan 'employee'SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]ROW                                      COLUMN+CELL                                                                                                            1                                       column=info:name, timestamp=1417426628685, value=michael                                                               2                                       column=info:name, timestamp=1417426628685, value=ted                                                                   3                                       column=info:name, timestamp=1417426628685, value=jack                                                                 3 row(s) in 0.1630 seconds

3 Data Records inserted
Import hive from mysql

# sqoop import --connect jdbc:mysql://localhost:3306/sqoop_test --username root --password root --table employee --hive-import --hive-table hive_employee --create-hive-tableWarning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.……………………14/12/02 15:12:13 INFO hive.HiveImport: Loading data to table default.hive_employee14/12/02 15:12:14 INFO hive.HiveImport: Table default.hive_employee stats: [num_partitions: 0, num_files: 4, num_rows: 0, total_size: 23, raw_data_size: 0]14/12/02 15:12:14 INFO hive.HiveImport: OK14/12/02 15:12:14 INFO hive.HiveImport: Time taken: 0.799 seconds14/12/02 15:12:14 INFO hive.HiveImport: Hive import complete.14/12/02 15:12:14 INFO hive.HiveImport: Export directory is empty, removing it.

In the real environment, do not use localhost for the jdbc link of mysql, because this task will be distributed and sent to different hadoop hosts. Only those hosts can be connected to mysql through jdbc, otherwise, data will be lost.
Check hive

hive> select * from hive_employee;OK1michael2ted3jackTime taken: 0.179 seconds, Fetched: 3 row(s)

Another point to note: currently sqoop can only import data from mysql to hive native tables (that is, hdfs-based storage ), data cannot be imported to external tables (for example, hive tables created based on hbase)
Class! Export next time!