SQOOP2 Environment Construction

Preparing to do the part of Spark SQL external data source interacting with the relational database, refer to how SQOOP2 operates the relational database.

: http://archive.cloudera.com/cdh5/cdh/5/

Download and install:

cd/home/spark/app/wget http://archive.cloudera.com/cdh5/cdh/5/ sqoop2-1.99.3-cdh5.0.0.tar.gztar -zxvf sqoop2-1.99. 3-cdh5. 0.0. Tar . GZCD sqoop2-1.99. 3-cdh5. 0.0

Note: Due to the local use of Hadoop is the 2.3.0-cdh5.0.0 version, so in this case, the use of cdh5.0.0 corresponding Sqoop version;

Add SQOOP2 to the system environment variable:

Export sqoop2_home=/home/spark/app/sqoop2-1.99. 3-cdh5. 0.0 export Catalina_base= $SQOOP 2_home/serverexport PATH=.: $SQOOP 2_home/bin: $PATH

Copy the MySQL driver package to $sqoop2_home/server/lib

CP mysql-connector-java-5.1. Ten

To modify a configuration file:

$SQOOP 2_home/server/conf/sqoop.properties

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/spark/app/ Hadoop-2.3.0-cdh5.0.0/etc/hadoop

$SQOOP 2_home/server/conf/catalina.properties

Common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar, ${catalina.home}/. /lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/common/*.jar,/home/spark/app/ hadoop-2.3.0-cdh5.0.0/share/hadoop/common/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/hdfs/*. jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/hdfs/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/ share/hadoop/mapreduce/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/mapreduce/lib/*.jar,/home/ spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/tools/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/ tools/lib/*.jar,/home/spark/app/hadoop-2.3.0-cdh5.0.0/share/hadoop/yarn/*.jar,/home/spark/app/ Hadoop-2.3.0-cdh5.0.0/share/hadoop/yarn/lib/*.jar

If you want to modify the tomcat port number and other information can be set in $sqoop2_home/server/conf/server.xml;

Start-Stop Sqoop-server:

$SQOOP 2_home/bin/sqoop. SH server Start$sqoop2_home/bin/sqoop. SH Server stop

Verify that the startup was successful: JPS: View process: Bootstrap
Way two: Http://hadoop000:12000/sqoop/version

To start the Sqoop client:

$SQOOP 2_home/bin/sqoop. SH Client

To configure the server for the client:

Sqoop:12000 --webapp Sqoop

To view server-side information:

Sqoop:> Show Server--all

SQOOP2 Environment Construction