Sqoop installation configuration and data import and export

Source: Internet
Author: User
Tags sqoop

front-facing conditions

The configuration of the Hadoop and MySQL database servers has been successfully installed, and if you import the data or export it from hbase, you should also have successfully installed HBase.

Download the JDBC driver for sqoop and MySQL

sqoop-1.2.0-cdh3b4.tar.gz :http://archive.cloudera.com/cdh/3/sqoop-1.2.0-CDH3B4.tar.gz

mysql-connector-java-5.1.28

Installing Sqoop

[Email protected] ~]$ tar -zxvf sqoop-1.2.0-cdh3b4.tar.gz

Configuring environment Variables

Copy Hadoop Core Packages and MYSQL Drive Package to Sqoop of the Lib Catalogue

[email protected] ~]$ cp Hadoop-1.1.2/hadoop-core-1.1.2.jar sqoop-1.2.0-cdh3b4/lib/

[email protected] ~]$ cp Mysql-connector-java-5.1.28-bin.jar sqoop-1.2.0-cdh3b4/lib/

Configuring the Sqoop-1.2.0-cdh3b4/bin/configure-sqoop file

Comment out HBase and zookeeper checks (unless you are ready to use Hadoop components such as habase)

start a Hadoop cluster

start MySQL

Create sqoop user

Build the Sqoop Library, test table, and construct the testing data

test Sqoop Connection

[Email protected] ~]$ sqoop list-databases--connect jdbc:mysql://10.120.10.11:3306/--username sqoop--password sqoop

Lists the names of all databases in MySQL

import from MySQL to HDFs

Sqoop # #sqoop command

Import # # indicates an imported

--connect Jdbc:mysql://ip:3306/sqoop # # tell jdbc, connect MySQL URL

--username Sqoop # # user name to connect to MySQL

--password Sqoop # # connection mysql password

--table Test # # # name of the table exported from MySQL

--fields-terminated-by ' \ t ' # # Specify the field delimiter for the rows in the output file

-M 1 # # 1 map jobs used in the copy process

[Email protected] ~]$ sqoop import--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--fields-terminated-by ': '-M 1

View import results in Hadoop

export from HDFs to MySQL

Sqoop

Export # # means data is copied from hive to MySQL

--connect Jdbc:mysql://ip:3306/sqoop

--username Sqoop

--password Sqoop

--table Test # #mysql table, the name of the table that will be imported

--export-dir '/user/root/aa/part-m-00000 ' # files exported in #hive

--fields-terminated-by ' \ t ' # #hive the delimiter of the exported file field

[Email protected] ~]$ sqoop export--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--export-dir '/user/hadoop/test/part-m-00000 '--fields-terminated-by ': '-M 1

view export results in Mysql

import from MySQL to Hbase

Parameter description:

Øhbase_tablename Specifies the name of the table to be converted to HBase

Økey_col_name specifies which column in the MySQL database table is the rowkey of the hbase new table

Øcol_fam_name is the column family name for all columns except Rowkey

[Email protected] ~]$ sqoop import--connect jdbc:mysql://10.120.10.11:3306/sqoop--username sqoop--password sqoop--ta ble test--hbase-create-table--hbase-table mysql_sqoop_test--column-family info--hbase-row-key id-m 1

viewing results in HBase

Transferred from: http://www.blogjava.net/redhatlinux/archive/2014/05/31/414291.html

Sqoop installation configuration and data import and export

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.