Using Sqoop2 to import mysql data to HDFS

標籤：sqoop2

On Sqoop2 server:
MySQL JDBC Driver:
cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/

Oracle JDBC Driver:
cp ojdbc6.jar /var/lib/sqoop2/

Microsoft SQL Server JDBC Driver:
cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/

cd /var/lib/sqoop2
chown sqoop2:sqoop2 mysql-connector-java-5.1.33-bin.jar ojdbc6.jar sqljdbc4.jar
chmod 755 mysql-connector-java-5.1.33-bin.jar ojdbc6.jar sqljdbc4.jar

restart sqoop2 service


On Sqoop2 client:
MySQL JDBC Driver:
cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/

Oracle JDBC Driver:
cp ojdbc6.jar /var/lib/sqoop2/

Microsoft SQL Server JDBC Driver:
cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/

cd /var/lib/sqoop2
chown sqoop2:sqoop2 mysql-connector-java-5.1.33-bin.jar ojdbc6.jar sqljdbc4.jar
chmod 755 mysql-connector-java-5.1.33-bin.jar ojdbc6.jar sqljdbc4.jar


sqoop2 (you can also use HUE web UI > sqoop transfer)
sqoop:000> set server --host sqoop2-server-ip
sqoop:000> show connector --all

sqoop:000> create connection --cid 1
Creating connection for connector with id 1
Please fill following values to create new connection object
Name: mysql-conection

Connection configuration

JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname      
Username: username
Password: *******
JDBC Connection Properties:
There are currently 0 values in the map:
entry#

Security related configuration options

Max connections: 10
New connection was successfully created with validation status FINE and persistent id 1

sqoop:000> create job --xid 1 --type import
Creating job for connection with id 1
Please fill following values to create new job object
Name: export-job1

Database configuration

Schema name: dbname
Table name: tablename
Table SQL statement:
Table column names:
Partition column name:
Nulls in partition column:
Boundary query:

Output configuration

Storage type:
  0 : HDFS
Choose: 0
Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 0
Compression format:
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
Choose: 0
Output directory: /user/sqoop2/output

Throttling resources

Extractors:
Loaders:
New job was successfully created with validation status FINE  and persistent id 1

sqoop:000> start job --jid 1

sqoop:000> status job --jid 1



JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname


JDBC Driver Class: com.microsoft.sqlserver.jdbc.SQLServerDriver
JDBC Connection String: jdbc:sqlserver://remote-mysql-server/dbname


JDBC Driver Class: oracle.jdbc.driver.OracleDriver
JDBC Connection String:

本文出自 “Ilovecat(個人筆記)” 部落格，請務必保留此出處http://hj192837.blog.51cto.com/655995/1570316

Using Sqoop2 to import mysql data to HDFS