Sqoop Study notes _sqoop Basic use of a

Sqoop

Relational DB and Hive/hdfs/hbase import the exported MapReduce framework.

Http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.4-cdh5.1.0/SqoopUserGuide.html

Etl:extraction-transformation-loading abbreviations, data extraction, transformations (business processing), and loading.

File data Source: Hive load Command

Relational DB data Source: Sqoop Extraction

Sqoop Import data to hdfs/hive/hbase--> Business processing--->sqoop export data to a relational database

Sqoop Import extracting from a relational database to HDFS

Sqoop Export from HDFS back to relational data

Second, Sqoop Development steps under Linux

Mysql Jdbc the drive is placed Sqoop_home/lib under

Put Lib directory (Eg:cd/opt/cloudera/parcels/chd/lib/sqoop/lib, sometimes both master and slave nodes to hold this lib)

1. Sqoop Basic Import and Export instruction (command line mode)

Sqoop import--connect jdbc:mysql://master:3306/test \ (This sentence is to configure the MySQL path, the next line of Linux is not completed to add a backslash)

--username root--password 123456--table rpt_sale_daily \ (Configure MySQL account and password and table table name)

--columns "DATEID,HUODONG,PV,UV" \ (Configure table-extracted fields)

--where "dateid= ' 2015-08-28 '" \ (table extraction query condition)

--target-dir sqoop/rpt_sale_daily \ (table extraction holds target location)

-M 1 (Sqoop extract the number of maps to be started, if the number of extracts can be adjusted to the amount of map)

Sqoop Import (extraction) needs to be noted:

Extract to Target-dir, column separated by commas;

When extracted to a hive table, it is automatically matched based on the column delimiter of the hive table.

2. Sqoop--options can also be used for sqoop extraction but does not support data transmission parameters

Sqoop--options-file./test.opt does not support sending parameters to opt files. (Shell script mode)

Write the execution file first test.opt

Write the Execute sqoop execute opt file command in the shell script.

Execute command

This is better than command line, because the shell script can be used to pass the argument. But it's still too much trouble.

Sqoop Study notes _sqoop Basic use of a