Hbase integrates hive and sqoop to export data to mysql and hbasesqoop.
Two advantages of hive integration of hbase tables:
1. import data to MYSQL.
2. convert an hbase table to another hbase table.
Three steps:
1. Associate hive with hbase as an external table:
SQL code
- Create external table hive_device_app (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)
- Stored by 'org. apache. hadoop. hive. hbase. hbasestoragehandler'
- With serdeproperties ("hbase. columns. mapping "=": key, cf: genera_type, cf: install_type, cf: label, cf: meid, cf: model, cf: pkg_name, cf: specific_type ")
- TBLPROPERTIES ("hbase. table. name" = "tb_yl_device_app_info1 ");
2. hbase is associated with hive and hive insertion and update operations directly affect data in hbase.
SQL code
- Create table hbase_device_app (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)
- Stored by 'org. apache. hadoop. hive. hbase. hbasestoragehandler'
- With serdeproperties ("hbase. columns. mapping "=": key, cf: genera_type, cf: install_type, cf: label, cf: meid, cf: model, cf: pkg_name, cf: specific_type ")
- TBLPROPERTIES ("hbase. table. name" = "tb_yl_device_app_info2 ");
3. Create a hive table
SQL code
- Create table hive_device_app_real (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)
4. Import External table data to hive real table
SQL code
- Insert overwrite table hive_device_app_real select * from hive_device_app
5. sqoop export hive data to mysql
SQL code
- Sqoop export -- connect jdbc: mysql: // Hadoop48/toplists-m 1 -- table hive_device_app_real -- export-dir/user/hive/warehouse/hive_device_app_real/1_00_0 -- input-null-string "\\\ N" -- input -null-non-string "\\\\ N" -- input-fields-terminated-by "\ 01" -- input-lines-terminated-by "\ n"
6. You can use hive built-in functions to process data from one table to another.
SQL code
- Insert overwrite table another_hive_hbase_related_table select * from hbase_device_app
To export data from hbase to mysql, follow these steps: 1345
One table in hbase is added to another table (hive built-in functions can be used in the middle for data processing): 226
Can hbase imported by sqoop be used directly? Or do I need to see the table in hbase and insert the imported data?
A simple command is enough.
Sqoop import -- jar-file (your jar package) -- class-name (classname) -- connect (mysql instance) -- username (username) -- password (password) -table (table name)-hbase-table (hbase table name)-column-family (familyColumn)-hbase-row-key (hbase rowkey) -- hbase-create-table
Java ioioioexception: SQLException in nextKeyValue is reported when data in the mysql database is imported to hive through sqoop.
Try to modify the SQL. Other SQL statements with the same effect