Oozie3.3.2 installation and deployment in hadoop2.2.0 and troubleshooting

Oozie's installation, deployment, and configuration are not complicated, and an endless number of bugs have overwhelmed me. The following describes the installation process and problems I encountered. prerequisites: 1.apache's tomcat2.mysql-connector-java-5.1.6.jaror its other mysql-connectorjarpackage 3.ext-2.2.zip II. installation Department

Oozie's installation, deployment, and configuration are not complicated, and an endless number of bugs have overwhelmed me. The following describes the installation process and problems I encountered. prerequisites: 1. apache tomcat 2. mysql-connector-java-5.1.6.jar or another version of mysql-connectorjar package 3.ext-2.2.zip II. installation Department

Oozie's installation, deployment, and configuration are not complicated, and the endless bugs make me feel overwhelmed. The following describes the installation process and problems I encountered.

I. necessary preparations:

1. Apache tomcat

2. mysql-connector-java-5.1.6.jar or other mysql-connectorjar packages

3. ext-2.2.zip

Ii. installation and deployment

0. Download the tar package for the oozie-3.3.2 and unzip: tar-xvf oozie-3.3.2.tar. Oozie4.0.0 Installation failed, the reason is that the oozie-setup.sh has been unable to execute, hope that the Daniel enlighten me.

1. In the decompressed directory, modify the hadoop version in pom. xml for the version you need, my hadoop-2.2.0, this step is because oozie needs to compile according to the hadoop version

2. Modify the hadoop version in the pom. xml file under the corresponding version directory of hadooplib. The lib generated in this directory will be used soon.

3. Run bin/mkdistro. sh-DskipTests to compile

4. after compilation, you can find the oozie-3.3.2-distro under OOZIE_HOME/distro/target/, This is the compiled version, our deployment is to deploy oozie under this directory, if there is no special instructions, the installation directory also refers to this directory.

5. In the oozie installation directory, extract the following tar packages: share, example (optional), and clint:

Oozie-client-3.3.2.tar.gz

Oozie-examples.tar.gz

Oozie-sharelib-3.3.2.tar.gz

6. decompress the share tar package and upload it to oozie of hdfs:

Hadoop fs-put share

7. create the libext folder under the oozie installation directory, find the corresponding version under the hadooplib directory in step 2, and find the corresponding lib directory under the target directory, copy all the jar packages to the new libext.

8. Add mysql-related packages:

Bin/oozie-setup.sh-jars mysql-connector-java-5.1.6.jar-extjs/ext-2.2.zip

Note that do not write an error in the jar package path of mysql-connector and ext.

9. Add hadoop-related packages

$ OOZIE_HOME/bin/oozie-setup.sh-hadoop 2.2.0 $ {HADOOP_PREFIX}-extjs/ext-2.2.zip

10. copy the mysql-connector jar package to lib and libext.

11. Environment Variables

Etc/profile: mainly set oozie and apache, as follows:

Export OOZIE_HOME =/usr/lib/cloud/oozie

Export CATALINA_HOME =/usr/lib/cloud/oozie/apache-tomcat-7.0.42

Export PATH = "$ {CATALINA_HOME}"/bin: $ PATH

Export OOZIE_URL = http: // localhost: 11000/oozie

Export PATH = "$ {OOZIE_HOME}"/bin :$ {PATH}

12. Modify the oozie-site.Xml file: Add the following, it feels good to use the default directly, do not need to modify the oozie-site.xml

Oozie. service. JPAService. create. db. schema

True

13. Add the following in the hadoop core-site.xml, oozie is the user name, hadoop is

Group

Hadoop. proxyuser. oozie. hosts

192.168.237.128

Hadoop. proxyuser. oozie. groups

Hadoop

14. Under the conf/hadoop-conf directory in oozie, modify the core-site.xml file and add:

Yarn. resourcemanager. address

Baby6: 8032

Yarn. resourcemanager. schedager. address

Master: 8030

15. Now you can start, execute oozie-start.sh under bin

If it works normally after startup, you can see the following interface on the webpage, which indicates that it looks successful:

Ii. Common problems:

1. the following error is reported during running:

JA017: Unknown hadoop job [job_13800000946375_0004] associated with action [0000000-131111204232566-oozie-oozi-W @ mr-node]. Failing this action!

There are many variants, because oozie could not find the hadoop job. Previously running well on MR1 was easy to see when it was upgraded to hadoop 2.x. x. My solution is to put jobhistory configuration information in the conf/hadoop-conf/core-site.xml of oozie, of course you must first enable jobhistory.

Enabling hadoop jobhistory is $ HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver, and it is estimated that many users who upgraded from hadoop 1.0.4 like me will forget this step.

Put the following information into the conf/hadoop-conf/core-site.xml of oozie:

Mapreduce. jobhistory. webapp. address
Baby6: 19888


Mapreduce. jobhistory. intermediate-done-dir
/User/yarn/tmp


Mapreduce. jobhistory. done-dir
/User/yarn/done

I guess it's because the oozie-3.3.2 doesn't have enough support for yarn, or because I'm too weak.

2. The second point is also due to the upgrade of hadoop, which is the jobtracker problem. Jobtraker does not exist in hadoop 2.2.0, but oozie still needs this parameter, which is obviously a version compatibility issue.

The solution is simple. Fill in the yarn. resourcemanager. address value for the jobtracker value. The default value is baby: 8032, and baby is your host name. In general, you need to change this value in workflow or job. properties, and write yarn. resourcemanager. address to the conf/hadoop-conf/core-site.xml of oozie, as mentioned in the above 14th