Hive Learning seven "Sqoop Import from relational database extraction to HDFs"

First, what is Sqoop

Sqoop is an open source tool that is used primarily in Hadoop (Hive) and traditional databases (MySQL, PostgreSQL ...) Data can be transferred from one relational database (such as MySQL, Oracle, Postgres, etc.) to the HDFs in Hadoop, or the data in HDFs can be directed into a relational database.

Second, the characteristics of Sqoop

One of the highlights of Sqoop is the ability to import data from a relational database into HDFs through the mapreduce of Hadoop.

Iii. Sqoop Import Command

Place the JDBC driver for MySQL in the /opt/cloudera/parcels/cdh/lib/sqoop/lib directory.

Such as:

　　

Reference Study Website:

　　Http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.4-cdh5.1.0/SqoopUserGuide.html

　　Detailed command:

1. Basic grammar

1 Import (Generic-args) (import-args) $ sqoop-import (Generic-args) (import-args)

2. Basic commands

View URL:

　　Http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.4-cdh5.1.0/SqoopUserGuide.html

　　Import control arguments for Sqoop import in

3. Case

　　Way One:

Import data from table rpt_sale_daily into HDFs.

The target directory is: sqoop/rpt_sale_daily.

Import --connect jdbc:mysql://master:3306/test \--username root--password 123456--  --columns "Dateid,huodong,pv,uv"--where "dateid= ' 2015-08-28 '"--target-dir sqoop/- M 1

If the prompt sqoop/rpt_sale_daily directory already exists, execute:

Hadoop FS-RMR sqoop/rpt_sale_daily

Execution Result: if

20 data successfully imported, start testing IT Pro.

Execute code:

Hadoop Fs-cat sqoop/rpt_sale_daily/part-m-00000

Data such as:

The data in the relational database table rpt_sale_daily is as follows:

Comparison, the data is the same, successfully imported.

Way Two:

Transfer via opt file:

New Test.opt File:

1 Import2--Connect3Jdbc:mysql://192.168.0.115:3306/test4--username5 Root6--Password7 123456 8--Table9 rpt_sale_dailyTen--Columns One "ID,HUODONG,PVV,UVV"  A--where  - "id= ' 2015-08-28 '"  ---target-dir thesqoop/rpt_sale_daily --M1

Execution Instructions:

Sqoop Options-file test.opt

The execution process is the same as the result and method.

Hive Learning seven "Sqoop Import from relational database extraction to HDFs"