Datax+hadoop2. x Compatibility debugging

Take Hdfsreader to Hdfswriter as an example to illustrate:

The 1.datax task profile needs to indicate which Hadoop configuration file is being used in DATAX+HADOOP1. x, you can use the HADOOP1 directly. X/conf/core-site.xml;

But when it's time to DATAX+HADOOP2. x, you need to HADOOP2. X/etc/core-site.xml and Hadoop2.x/etc/hdfs-site.xml synthesize a file and can be named Hadoop-site.xml.

2. In the synthesized hadoop-site.xml file, new attributes are required:

<property>
<name>Fs.hdfs.impl</name> <!--hdfsreader/hdfswriter dir is required to be incremented when hdfs://, which represents the HDFs path--
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>

<property>
<name>Fs.file.impl</name> <!--hdfsreader/hdfswriter dir is required to be added when file://, indicating local path---
<value>org.apache.hadoop.fs.LocalFileSystem</value>
</property>

3. Additional dependency packages are required for Hdfsreader, including:

-rw-r--r--1 Hadoop hadoop 575389 Dec 16:24 Commons-collections-3.2.1.jar

-rw-r--r--1 Hadoop hadoop 62050 Dec 16:23 Commons-logging-1.1.3.jar

-rw-r--r--1 Hadoop hadoop 1648200 Dec 16:25 Guava-11.0.2.jar
-rw-r--r--1 Hadoop hadoop 3318401 Dec 16:26 Hadoop-common-2.6.2.jar
-rw-r--r--1 Hadoop hadoop 178199 Dec 16:26 Hadoop-lzo-0.4.20-snapshot.jar
-rw-r--r--1 Hadoop hadoop 16380 Dec 15:29 Hdfsreader-1.0.0.jar
-rw-r--r--1 Hadoop hadoop 18490 Dec 15:29 Java-xmlbuilder-0.4.jar
-rw-r--r--1 Hadoop Hadoop 2019 Dec 15:29 Paramkey.java
-rwxr-xr-x 1 Hadoop hadoop 18837 Dec 15:29 Plugins-common-1.0.0.jar

Hdfsread/hadoop-0.19.2-core.jar (Hadoop*-core*.jar) needs to be removed.

4. Additional dependency packages are required for hdfswriter, including:

-rwxr-xr-x 1 Hadoop hadoop 41123 Dec 16:40 Commons-cli-1.2.jar

-rw-r--r--1 Hadoop hadoop 575389 Dec 16:34 Commons-collections-3.2.1.jar
-rw-r--r--1 Hadoop hadoop 62050 Dec 16:34 Commons-logging-1.1.3.jar
-rw-r--r--1 Hadoop hadoop 1648200 Dec 16:34 Guava-11.0.2.jar
-rwxr-xr-x 1 Hadoop hadoop 67190 Dec 16:40 Hadoop-auth-2.6.2.jar
-rw-r--r--1 Hadoop hadoop 3318401 Dec 16:34 Hadoop-common-2.6.2.jar
-rwxr-xr-x 1 Hadoop hadoop 7915385 Dec 16:36 Hadoop-hdfs-2.6.2.jar
-rw-r--r--1 Hadoop hadoop 178199 Dec 16:34 Hadoop-lzo-0.4.20-snapshot.jar
-rw-r--r--1 Hadoop hadoop 14652 Dec 16:35 Hdfswriter-1.0.0.jar
-rwxr-xr-x 1 Hadoop hadoop 31212 Dec 16:43 Htrace-core-3.0.4.jar
-rw-r--r--1 Hadoop hadoop 18490 Dec 16:34 Java-xmlbuilder-0.4.jar
-rw-r--r--1 Hadoop hadoop 657766 Dec 15:28 libhadoop.so
-rw-r--r--1 Hadoop hadoop 4374 Dec 15:28 Paramkey.java
-rwxr-xr-x 1 Hadoop hadoop 18837 Dec 16:34 Plugins-common-1.0.0.jar
-rwxr-xr-x 1 Hadoop hadoop 533455 Dec 16:43 Protobuf-java-2.5.0.jar

Hdfsread/hadoop-0.19.2-core.jar (Hadoop*-core*.jar) needs to be removed.

5. Environment variables must be properly configured, such as:

Path= $PATH: $HOME/app/bin. errors, which are difficult to spot and cause problems

Path= $PATH: $HOME/app/bin:.://correct, current directory to be used separately: separate

Datax+hadoop2. x Compatibility debugging