Sqoop2從Mysql匯入Hdfs (hadoop-2.7.1,Sqoop 1.99.6)

標籤：

一、環境搭建

1.Hadoop

http://my.oschina.net/u/204498/blog/519789

2.Sqoop2.x

http://my.oschina.net/u/204498/blog/518941

3. mysql

二、從mysql匯入hdfs

1.建立mysql資料庫、表、以及測試資料

xxxxxxxx$  mysql -uroot -pEnter password: mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || mysql              || performance_schema || test               |+--------------------+4 rows in set (0.00 sec)test  => 是建立的資料庫mysql> use test;mysql> show tables;+----------------------+| Tables_in_test       |+----------------------+               || test                 |+----------------------+1 rows in set (0.00 sec)test => 是新增的表mysql> desc test;+-------+-------------+------+-----+---------+----------------+| Field | Type        | Null | Key | Default | Extra          |+-------+-------------+------+-----+---------+----------------+| id    | int(11)     | NO   | PRI | NULL    | auto_increment || name  | varchar(45) | YES  |     | NULL    |                || age   | int(11)     | YES  |     | NULL    |                |+-------+-------------+------+-----+---------+----------------+3 rows in set (0.00 sec)mysql> select * from test;+----+------+------+| id | name | age  |+----+------+------+|  7 | a    |    1 ||  8 | b    |    2 ||  9 | c    |    3 |+----+------+------+3 rows in set (0.00 sec)

2. 為各個使用者授權

注意：sqoop提交job後，各個節點在map階段會訪問資料庫，所以需事先授權

mysql> grant [all | select | ...] on {db}.{table} to {user}@{host} identified by {passwd};mysql> flush privileges;#我給特定的hostname授權 username:root passwd:root 訪問db:test 中任意table,許可權是allmysql> grant all on test.* to ‘root‘@{host} identified by ‘root‘;

3.啟動sqoop2-server

[[email protected] sqoop-1.99.6-bin-hadoop200]$ pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200[[email protected] sqoop-1.99.6-bin-hadoop200]$ ./bin/sqoop2-server start......webui可以訪問校正，也可以查看log

4.啟動sqoop2-shell

[[email protected] sqoop-1.99.6-bin-hadoop200]$ pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200[[email protected] sqoop-1.99.6-bin-hadoop200]$ ./bin/sqoop2-shell ......sqoop:000> show version......sqoop:000> show connector+----+------------------------+---------+------------------------------------------------------+----------------------+| Id |          Name          | Version |                        Class                         | Supported Directions |+----+------------------------+---------+------------------------------------------------------+----------------------+| 1  | generic-jdbc-connector | 1.99.6  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO              || 2  | kite-connector         | 1.99.6  | org.apache.sqoop.connector.kite.KiteConnector        | FROM/TO              || 3  | hdfs-connector         | 1.99.6  | org.apache.sqoop.connector.hdfs.HdfsConnector        | FROM/TO              || 4  | kafka-connector        | 1.99.6  | org.apache.sqoop.connector.kafka.KafkaConnector      | TO                   |+----+------------------------+---------+------------------------------------------------------+----------------------+根據你的connector建立connectorsqoop:000> create link -c 1      => 先建立jdbc會填寫name、jdbc-driver、url、username、passwd等等sqoop:000> create link -c 3      => 建立hdfs會填寫name、hdfs url、等等sqoop:000> show link+----+-------------+--------------+------------------------+---------+| Id |    Name     | Connector Id |     Connector Name     | Enabled |+----+-------------+--------------+------------------------+---------+| 3  | 10-21_jdbc1 | 1            | generic-jdbc-connector | true    || 4  | 10-21_hdfs1 | 3            | hdfs-connector         | true    |+----+-------------+--------------+------------------------+---------+建立job -f=> from  -t to 即從哪些匯入到哪裡sqoop:000> create job -f 3 -t 4會填寫，相應的table資訊。還有hdfs資訊sqoop:000> show job             +----+---------------+----------------+--------------+---------+| Id |     Name      | From Connector | To Connector | Enabled |+----+---------------+----------------+--------------+---------+| 1  | 10-20_sqoopy2 | 1              | 3            | true    |+----+---------------+----------------+--------------+---------+#啟動jobsqoop:000> start job -j 2.........可以再webui上訪問到，查看進度，也可以使用sqoop:000> status job -j 2

sqoop的guide

http://sqoop.apache.org/

5.troubleshooting

多看日誌，慢慢的排查

Sqoop2從Mysql匯入Hdfs (hadoop-2.7.1,Sqoop 1.99.6)