Sqoop2 installation record

I used the source code compilation package for installation, mainly considering that some features will be cropped or enhanced. For details about the source code compilation method, refer to another blog 《Solve sqoop2 compilation errors. Then, the content of the beibeisqoop-1.99.3.tar.gz compressed package under the DIST/targetdirectory is included in the/usr/lib/sqoop directory.

The next step is to configure the relevant configuration files. Note that sqoop2 distinguishes the server from the client. First, install the server. Remember that the server must be installed on a machine that can execute the hadoop command line, the most direct method is to install it on a hadoop cluster node, and the client can communicate with sqoop server on any machine.

1. Server installation and configuration

Go to the/usr/lib/sqoop/Server/conf directory. You can see several configuration files in the directory. For the Tomcat container, there are sqoop. First, we configure Catilina. properties: The jar package of hadoop must be dependent. Of course, if you want to import hive or hbase, add the corresponding jar package as follows:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/common/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/common/lib/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/hdfs/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/hdfs/lib/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/mapreduce/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/mapreduce/lib/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/tools/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/tools/lib/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/yarn/*.jar,/home/cdh/hadoop-2.3.0-cdh5.1.2/share/hadoop/yarn/lib/*.jar

Then modify sqoop. properties, which can be used to configure the log output path and Deby information of the metadata embedded database. You must note that the Deby database name is changed to sqoop. This is not the default one, but it is written to the code, modify:

<pre name="code" class="html"># JDBC repository provider configurationorg.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandlerorg.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTEDorg.apache.sqoop.repository.jdbc.maximum.connections=10org.apache.sqoop.repository.jdbc.url=jdbc:derby:@[email protected]/repository/SQOOP;create=trueorg.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriverorg.apache.sqoop.repository.jdbc.user=saorg.apache.sqoop.repository.jdbc.password=

## Configuration for Mapreduce submission engine (applicable if it's configured)## Hadoop configuration directoryorg.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/cdh/hadoop/etc/hadoop/

In addition, you do not need to configure special configurations for the time being. You only need to configure them by default. After all, it is important to start playing first!

The last thing to do is to put the MySQL driver of the mysql-connector-java-5.1.20.jar into the/usr/lib/sqoop/Server/lib directory:

The last step is to start the server (I installed it on 192.168.69.16 ):

/Usr/lib/sqoop/bin/sqoop. Sh Server start

As a client without any configuration, the compressed distribution package is directly sent to the corresponding machine for decompression. Run bin/sqoop. Sh client to enter the shell command line interface. For details, refer to the official documentation:

《Command line Client"

For legacy problems, an error is always reported during startup. It should be caused by a log jar conflict and cannot be solved. If you know how to handle the problem, you can share the following information:

log4j: Finished configuring.log4j:ERROR A "org.apache.log4j.xml.DOMConfigurator" object is not assignable to a "org.apache.log4j.spi.Configurator" variable.log4j:ERROR The class "org.apache.log4j.spi.Configurator" was loaded bylog4j:ERROR [[email protected]] whereas object of typelog4j:ERROR "org.apache.log4j.xml.DOMConfigurator" was loaded by [WebappClassLoader^M  context: /sqoop^M   delegate: false^M   repositories:^M    /WEB-INF/classes/^M ----------> Parent Classloader:^M[email protected]^M].  log4j:ERROR Could not instantiate configurator [org.apache.log4j.xml.DOMConfigurator].log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.

Sqoop2 installation record