Sqoop Sample Code

The database used in this article is MySQL's sample DB employees.

Download url:https://launchpad.net/test-db/employees-db-1/1.0.6

Then install to your MySQL database according to the README.

Installation of Sqoop:

: http://apache.dataguru.cn/sqoop/1.4.6/

Sqoop-1.4.6.tar.gz                 

My Hadoop version is 2.6, so I downloaded these two versions.

Sqoop-1.4.6.tar.gz cannot be used directly and needs to be compiled. After compiling the generated

Sqoop-1.4.6.jar Copy to Sqoop_home/lib below.

Replicating JDBC Connector

Mysql-connector-java-5.1.32-bin.jar or a higher version of MySQL JDBC connector.

Otherwise, some bugs will appear.

Sqoop list-tables--connect Jdbc:mysql://namenode01:3306/employees--username hive--password HiveMysqlurl=jdbc:mysql://namenode01:3306/employeesSqoop Import--Connect $mysqlurl--username Hive--Password Hive--Table Departments--target-dir/etl/input/Departments HDFs DFS-Cat/etl/input/departments/* INSERT into departments values (' d9999 ', ' evan,test '); overriding Type Mapping--map-column-java c1=float,c2=string,c3= String ... Sqoop By default uses four concurrent maps tasks to transfer data to Hadoop. Mysqlurl=jdbc:mysql://namenode01:3306/employees sqoop Import--connect $mysqlurl--username hive--password Hive-- Query ' Select D.dept_no,d.dept_name,de.from_date, De.to_date, e.* from Employees e join Dept_emp de on e.emp_no=de.emp_no Join Departments D on De.dept_no=d.dept_no WHERE $CONDITIONS '--split-by d.dept_no--target-dir/etl/input/employees--co Mpare the result count from MySQL and HDFs Filehdfs dfs-cat/etl/input/employees/* | Wc-lif want to overwrite the data type, you can specify the column=type--map-column-hive id=string,price=decimal mysq Lurl=jdbc:mysql://namenode01:3306/employees sqoop Import--connect $mysqlurl--username hive--password Hive--query ' Select D.dept_no,d.dept_name,de.from_date, De.to_date, e.* from Employees e join Dept_emp de on e.emp_no=de.emp_no Join Departments D on De.dept_no=d.dept_no WHERE $CONDITIONS '--split-by d.dept_no--hive-import--hive-table Test.employ EES--hive-drop-import-delims--null-string ' \\n '--null-non-string ' \\n '--target-dir/tmp/employees

Sqoop Sample Code