Hbase integrates hive and sqoop to export data to mysql and hbasesqoop.

Source: Internet
Author: User
Tags sqoop

Hbase integrates hive and sqoop to export data to mysql and hbasesqoop.
Two advantages of hive integration of hbase tables: 
1. import data to MYSQL.
2. convert an hbase table to another hbase table.
 


Three steps:
1. Associate hive with hbase as an external table:
SQL code

  1. Create external table hive_device_app (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)
  2. Stored by 'org. apache. hadoop. hive. hbase. hbasestoragehandler'
  3. With serdeproperties ("hbase. columns. mapping "=": key, cf: genera_type, cf: install_type, cf: label, cf: meid, cf: model, cf: pkg_name, cf: specific_type ")
  4. TBLPROPERTIES ("hbase. table. name" = "tb_yl_device_app_info1 ");


2. hbase is associated with hive and hive insertion and update operations directly affect data in hbase.
SQL code
  1. Create table hbase_device_app (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)
  2. Stored by 'org. apache. hadoop. hive. hbase. hbasestoragehandler'
  3. With serdeproperties ("hbase. columns. mapping "=": key, cf: genera_type, cf: install_type, cf: label, cf: meid, cf: model, cf: pkg_name, cf: specific_type ")
  4. TBLPROPERTIES ("hbase. table. name" = "tb_yl_device_app_info2 ");


3. Create a hive table
SQL code
  1. Create table hive_device_app_real (row_key string, genera_type string, install_type string, label string, meid string, model string, pkg_name string, specific_type string)


4. Import External table data to hive real table
SQL code
  1. Insert overwrite table hive_device_app_real select * from hive_device_app

5. sqoop export hive data to mysql
SQL code
  1. Sqoop export -- connect jdbc: mysql: // Hadoop48/toplists-m 1 -- table hive_device_app_real -- export-dir/user/hive/warehouse/hive_device_app_real/1_00_0 -- input-null-string "\\\ N" -- input -null-non-string "\\\\ N" -- input-fields-terminated-by "\ 01" -- input-lines-terminated-by "\ n"


6. You can use hive built-in functions to process data from one table to another.
SQL code
  1. Insert overwrite table another_hive_hbase_related_table select * from hbase_device_app



To export data from hbase to mysql, follow these steps: 1345
One table in hbase is added to another table (hive built-in functions can be used in the middle for data processing): 226
 
Can hbase imported by sqoop be used directly? Or do I need to see the table in hbase and insert the imported data?

A simple command is enough.

Sqoop import -- jar-file (your jar package) -- class-name (classname) -- connect (mysql instance) -- username (username) -- password (password) -table (table name)-hbase-table (hbase table name)-column-family (familyColumn)-hbase-row-key (hbase rowkey) -- hbase-create-table

Java ioioioexception: SQLException in nextKeyValue is reported when data in the mysql database is imported to hive through sqoop.

Try to modify the SQL. Other SQL statements with the same effect

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.