SQOOP2 importing HDFs from MySQL (Hadoop-2.7.1,sqoop 1.99.6)

First, Environment construction

1.Hadoop

http://my.oschina.net/u/204498/blog/519789

2.sqoop2.x

http://my.oschina.net/u/204498/blog/518941

3. mysql

Second, import HDFs from MySQL

1. Create MySQL database, table, and test data

xxxxxxxx$  mysql -uroot -penter password: mysql> show databases;+-- ------------------+| database           |+-------- ------------+| information_schema | |  mysql              | |  performance_schema | |  test               |+-------- ------------+4 rows in set  (0.00 sec) test  =>  is a new database mysql>  use test;mysql> show tables;+----------------------+| tables_in_test        |+----------------------+                | |  test                  |+----------------------+1 rows in set  (0.00 SEC) test =>  is the new table mysql> desc test;+- ------+-------------+------+-----+---------+----------------+| field | type         | Null | Key | Default | Extra           |+-------+-------------+------+-----+---------+--------------- -+| id    | int (one)      | no   |  pri | null    | auto_increment | |  name  | varchar ( | yes  |     | null)     |                 | |  age   | int (one)      | YES  |      | null    |                 |+-------+-------------+------+-----+---------+----------------+3 rows in set  (0.00  SEC) mysql> select * from test;+----+------+------+| id | name |  age  |+----+------+------+|  7 | a    |     1 | |   8 | b    |    2 | |   9 | c    |    3 |+----+------+------+3  rows in set  (0.00 SEC)

2. Licensing for individual users

Note: After the Sqoop commits the job, each node accesses the database during the map phase, so the prior authorization is required

mysql> Grant [All | select | ...] on {db}. {table} to {User}@{host} identified by {passwd};mysql> flush privileges; #我给特定的hostname授权 username:root passwd:root To access any table in Db:test, the permission is allmysql> grant all on test.* to ' root ' @{host} identified by ' Root ';

3. Start Sqoop2-server

[Email protected] sqoop-1.99.6-bin-hadoop200]$ Pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200[[email protected] sqoop-1.99.6-bin-hadoop200]$./bin/sqoop2-server Start......webui can access the checksum, or you can view the log

4. Start Sqoop2-shell

[[email protected] sqoop-1.99.6-bin-hadoop200]$ pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200[[ email protected] sqoop-1.99.6-bin-hadoop200]$ ./bin/sqoop2-shell ......sqoop:000>  show version......sqoop:000> show connector+----+------------------------+---------+-------- ----------------------------------------------+----------------------+| id |           Name          |  version |                         Class                          |  supported directions |+----+------------------------+---------+------------------------------------- -----------------+----------------------+| 1  | generic-jdbc-connector | 1.99.6  |  org.apache.sqoop.connector.jdbc.genericjdbcconnector | from/to               | |  2  | kite-connector         | 1.99.6   | org.apache.sqoop.connector.kite.KiteConnector         | from/to              | |  3  | hdfs-connector         | 1.99.6   | org.apache.sqoop.connector.hdfs.HdfsConnector         | from/to              | |  4  | kafka-connector        | 1.99.6  | org.apache.sqoop.connector.kafka.KafkaConnector      |  to                    |+----+------------------------+---------+------------------------------------------------------+-------- --------------+ Create connectorsqoop:000> create link -c 1    based on your connector    =>  Create JDBC First fills in name, Jdbc-driver, URL, username, passwd, and so on sqoop:000> create  link -c 3      =>  Create HDFs will fill in name, Hdfs url, etc. sqoop:000>  show link+----+-------------+--------------+------------------------+---------+| id |     Name     | Connector Id |      connector name     | enabled |+----+-------------+-------- ------+------------------------+---------+| 3  | 10-21_jdbc1 | 1             | generic-jdbc-connector | true    | |  4  | 10-21_hdfs1 | 3             | hdfs-connector         | true     |+----+-------------+--------------+------------------------+---------+ Create job -f=>  from  -t to  from which to import to where sqoop:000> create job -f 3 -t  4 will fill in the corresponding table information. and HDFs Information sqoop:000> show job              +----+---------------+----------------+--------------+---------+| id |      name      | from connector | to connector  | enabled |+----+---------------+----------------+--------------+---------+| 1  | 10-20 _sqoopy2 | 1              |  3            | true     |+----+---------------+----------------+--------------+---------+ #启动jobsqoop: 000> start job  -j 2. You can webui petition again to check the progress, you can also use the SQOOP:000> STATUS JOB -J 2

Sqoop's Guide

http://sqoop.apache.org/

5.troubleshooting

Look at the log, slowly troubleshooting

SQOOP2 importing HDFs from MySQL (Hadoop-2.7.1,sqoop 1.99.6)