Sqoop synchronizing MySQL to HDFs

Link: Http://pan.baidu.com/s/1gfHnaVL Password: 7j12

Mysql-connector version 5.1.32

If you encounter some problems during the installation process, refer to http://dbspace.blog.51cto.com/6873717/1875955, some of which are the solution to the problem

Download and install:

cd/usr/local/
TAR-ZXVF sqoop2-1.99.3-cdh5.0.0.tar.gz
MV sqoop2-1.99.3-cdh5.0.0 Sqoop
Add SQOOP2 to the system environment variable:
Export Sqoop_home=/usr/local/sqoop
Export catalina_base= $SQOOP _home/server
Export path= $PATH:/usr/local/sqoop/bin
Copy the MySQL driver package to $sqoop2_home/server/lib
CP mysql-connector-java-5.1.32-bin.jar/usr/local/sqloop/server/lib/
To modify a configuration file:
vim/usr/local/sqoop/server/conf/sqoop.properties
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop#hadoop的配置文件路径
Vim/usr/local/sqoop/server/conf/catalina.properties
Note the original 58 lines, where the main configuration of the Hadoop jar package path information
Common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar, ${ca talina.home}/.    /lib/*.jar,/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*. jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/ Shar E/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop /tools/*. jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/ Local/hadoop/sh Are/hadoop/yarn/lib/*.jar
Start \ Stop Sqoop
/usr/local/sqoop/sqoop2-server start/stop
Verify that the startup was successful:
Way one: JPS View process: Bootstrap
[Email protected] sqoop]#JPS

25505 Sqoopshell
13080 Secondarynamenode
12878 NameNode
26568 Jps
Way two: Way two: Http://192.168.1.114:12000/sqoop/version #SQOOP默认使用的端口为12000在/usr/local/sqoop/server/conf/ setting in Server.xml

# # #接下来测试mysql到hadoop存储的一个过程
1. Client Login
[Email protected] bin]# Sqoop2-shell
Sqoop Home directory:/usr/local/sqoop
Sqoop shell:type ' help ' or ' \h ' for help.
Sqoop:000>
2, create a MySQL link, in this version create only [Connection|job], note that different versions of the way to add links are different.
View supported linked Services
Sqoop:000> Show Connector
+----+------------------------+-----------------+------------------------------------------------------+
|          Id |     Name |                        Version | Class |
+----+------------------------+-----------------+------------------------------------------------------+
| 1 | Generic-jdbc-connector | 1.99.3-cdh5.0.0 | Org.apache.sqoop.connector.jdbc.GenericJdbcConnector |
+----+------------------------+-----------------+------------------------------------------------------+# #在1.99.7 the way the version is displayed and service more.
sqoop:000> Create connection--cid 1
Creating connection for connector with ID 1
Please fill following values to create new Connection object
Name:mysql_to_hadoop

Connection Configuration

JDBC Driver Class:com.mysql.jdbc.Driver
JDBC Connection String:jDbc:mysql://192.168.1.107:3306/sqoop#这里需要在1.107 Add a good library first sqoop
Username:sqoop##需要在数据库添加好链接的用户
Password: *******
JDBC Connection Properties:
There is currently 0 values in the map:
entry#
Security Related configuration options
Max Connections:
New connection is successfully created with validation status acceptable and persistent ID 2
2. Create Job
Sqoop:000>Create job--xid 2--type import# #注意--xid 2 is the ID number of the link
Creating job for connection with ID 2
Please fill following the values to create new Job Object
Name:mysql_to_hadoop

Database Configuration

Schema Name:Sqoop#MySQL的库名
Table Name:wangyuan#Tables under the Library
Table SQL Statement:
Table column names:
Partition Column Name:
Nulls in partition column:
Boundary query:

Output Configuration

Storage Type:
0:hdfs
choose:0
Output format:
0:text_file
1:sequence_file
choose:0
Compression format:
0:none
1:default
2:deflate
3:gzip
4:bzip2
5:lzo
6:lz4
7:snappy
choose:0
Output directory:HDFS://192.168.1.114:9000/HOME/MYSQL_TO_HDFS2
#注意这个mysql_to_hdfs不能再hadoop的/home/already exists, but the/home path is to exist, port 9000 is configured when Hadoop is configured, depending on the actual, or through the Web to view http:ip:50070---- Show Overview ' mycat:9000 ' (active)
Create an HDFs path/usr/local/hadoop/bin/hadoop fs-mkidr/home
View Create directory:/usr/local/hadoop/bin/hadoop fs-ls/home or Web view http:ip:50070
Throttling Resources
Extractors:
Loaders:
New job is successfully created with validation status FINE and persistent ID 2
Sqoop:000>
Start Job
sqoop:000> Start Job--jid 2
Exception has occurred during processing command
Exception:org.apache.sqoop.common.SqoopException Message:CLIENT_0001:Server has returned Exception
Don't even know what this tip says by modifying the settings:
set option--name verbose--value true
Sqoop:000>Start Job--jid 2
Submission Details
Job Id:2
Server url:http://localhost:12000/sqoop/
Created By:root
Creation date:2016-11-23 21:15:27 CST
Lastly updated By:root
External id:job_1479653943050_0007
http://haproxy:8088/proxy/application_1479653943050_0007/
Connector schema:schema{name=wangyuan,columns=[
Fixedpoint{name=id,nullable=null,bytesize=null,unsigned=null},
Date{name=c_time,nullable=null,fraction=null,timezone=null}]}
2016-11-23 21:15:27 cst:booting-progress is not available
Return this information OK
View Results by web

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M01/8A/9E/wKioL1g1mbPR2z7uAABQbA5yHps688.png-wh_500x0-wm_3 -wmp_4-s_241985364.png "title=" 222.png "alt=" Wkiol1g1mbpr2z7uaabqba5yhps688.png-wh_50 "/>

/usr/local/hadoop/bin/hadoop fs-ls/home/

This article is from the "DBSpace" blog, so be sure to keep this source http://dbspace.blog.51cto.com/6873717/1875971

Sqoop synchronizing MySQL to HDFs