Configure oozie4.10+hadoop2.5.2

Finally, this mysterious looking person Oozie installation configuration succeeded, this troubled me for several days, when see the following screen, I think value!

Talk less, see how I compile and install the process:

(The HADOOP2.5.2HA environment has been set up, Hive,habase, Flume,stom have

Linux environment: centos6.5 64bit

jdk:1.7

MySQL has been installed

Apache Maven 3.1.1

Download Oozie installation package: oozie-4.1.0.tar.gz http://mirror.bit.edu.cn/apache/oozie/

Download Ext-2.2.zip http://oozie.apache.org/docs/4.0.1/DG_QuickStart.html This path has a ExtJS link

）

1. Compiling

Go to http://mirrors.cnnic.cn/apache/oozie/4.2.0/

The download is oozie-4.2.0.tar.gz

, then unzip:

TAR-ZXVF oozie-4.2.0.tar.gz

CD Oozie-4.2.0/bin

./mkdistro.sh-dskiptests-phadoop-2-dhadoop.auth.version=2.5.2-ddistcp.version=2.5.2-dsqoop.version=1.4.4- dhive.version=0.13.1

-dtomcat.version=7.0.52

A long wait, the network problem has been bothering me, I repeat the above command, found that, finally came here,

That's the wrong thing to compile here.

[INFO]------------------------------------------------------------------------[INFO] Reactor Summary:[info][info ] Apache Oozie Main ..... ....... ............... SUCCESS [6.824s][info] Apache Oozie Hadoop Utils hadoop-2-4.2.0 .... SUCCESS [9.525s][info] Apache Oozie Hadoop Distcp hadoop-2-4.2.0 ... SUCCESS [0.444s][info] Apache Oozie Hadoop Auth hadoop-2-4.2.0 Test ... SUCCESS [1.027s][info] Apache Oozie Hadoop Libs ......... ......... SUCCESS [0.101s][info] Apache Oozie Client ......... .............. SUCCESS [5:08.683s][info] Apache Oozie Share Lib Oozie ............. SUCCESS [9.351s][info] Apache Oozie Share Lib hcatalog ........ ..... SUCCESS [11.656s][info] Apache Oozie Share Lib Distcp ......... ..... SUCCESS [3.151s][info] Apache Oozie Core ..... ....... .............. SUCCESS [3:53.804s][info] Apache Oozie Share Lib streaming ....... ..... SUCCESS [13.230s][info] Apache Oozie Share Lib Pig ......... .......SUCCESS [15.454s][info] Apache Oozie Share Lib Hive ......... ....... SUCCESS [13.747s][info] Apache Oozie Share Lib Hive 2 ....... ....... SUCCESS [14.417s][info] Apache Oozie Share Lib Sqoop ............. SUCCESS [5.546s][info] Apache Oozie Examples ......... ............ SUCCESS [10.178s][info] Apache Oozie Share Lib Spark ........ ....... SUCCESS [15.450s][info] Apache Oozie Share Lib ......... ........... SUCCESS [52.422s][info] Apache Oozie Docs ...... ..... ............... FAILURE [9.477s][info] Apache Oozie WebApp ......... .............. Skipped[info] Apache Oozie Tools ..... ....... .............. Skipped[info] Apache Oozie minioozie ......... ........... Skipped[info] Apache Oozie distro ......... .............. Skipped[info] Apache Oozie ZooKeeper Security Tests .... ..... Skipped[info]------------------------------------------------------------------------[INFO] BUILD Failure[info]- -----------------------------------------------------------------------[INFO] Total time:12:21.113s[info] finished at:wed Oct 26 05:39:28 CST 2016[info] Final Memory:174m/482m[info]--------------------------------------------------------------                                                                                        ----------[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:2.0-be Ta-6:site (default) on Project Oozie-docs:the site descriptor can Not being resolve D from the Repositor                                                                                        Y:could Not transfer artifact Org.apache:apache:xml:site_en                                                                                        : From/to Codehaus Repository (http://repository.codehaus.org/): repository.co dehaus.org: Unknown name or service [ERROR] Org.apache:apache:xml: 16[error][error] from the specified rEmote Repositories:[error] Central (http://repo1.maven.org/maven2, Releases=true, Snapshots=false), [ERROR] CE D (https                                                                                        ://repository.cloudera.com/cloudera/ext-release-local/, Relea Ses=true, Snapshots=false), [ERROR] Codehaus repository (http://repository.codehaus.org/, Releas Es=true, SNA pshots=false), [ERROR]                                                                                        Cloudera COM (Https://repository.cloudera.com/content/repositories/relea ses/, Releases=true, Snapshots=false), [ERROR] central maven (http://central.maven.or                                                                                        g/maven2/, Releases=true, snapshot                                                             S=false), [ERROR] Apache.snapshots.repo (https://repository.apache.org/content/groups/snap                           Shots, Releases=true, snapshots=true), [ERROR] DataNucleus (http://www.datanucleus.org/downloads /maven2, Releases=true, snapshots=                                                                                        False), [ERROR] Apache.snapshots (http://repository.apache.org/snapshots, Releases=false , snapshots=true): Unknown host repository.codehaus.org: unknown name or service [ERR                                                                                        OR], [help 1][error][error] to see the full stack trace of the errors, re-run Maven with The-e Swit Ch.[error] Re-run Maven using the-x switch to Enab Le full debug logging. [ERROR] [ERROR]                                                                                        For more information on the errors and possible solutions, please REA D the following articles:[error] [Help 1] http://cwiki.apache.org/conflUence/display/maven/mojoexecutione  Xception[error][error] After correcting the problems, you can resume the build with the Command[error] mvn <goals> -rf:oozie-docserror, Oozie distro creation failed

　　

Helpless, and then went to compile, 3.3.2, the result:

It's not going to compile anymore.

Here, I go to the net to find the reason:

Told me it was: MAVEN's storage address, so I changed the storage configuration:

Oozie root directory,

Pom.xml, modify the storage in the:<repositories></repositories> as follows:

<repositories> <repository> <id>cloudera com</id> <url>https://repos Itory.cloudera.com/content/repositories/releases/</url> <snapshots> <enabled>f alse</enabled> </snapshots> </repository> <repository> <id&                Gt;central</id> <url>http://repo1.maven.org/maven2</url> <snapshots>            <enabled>false</enabled> </snapshots> </repository><repository>            <id>central maven</id> <url>http://central.maven.org/maven2/</url> <snapshots> <enabled>false</enabled> </snapshots> </repository > <repository> <id>codehaus repository</id> <url>http://repository . CODEHAUS.ORG/&LT;/URL&GT <snapshots> <enabled>false</enabled> </snapshots> </repository > <repository> <id>apache.snapshots.repo</id> <url>https://reposit            ory.apache.org/content/groups/snapshots</url> <name>apache Snapshots Repository</name> <snapshots> <enabled>true</enabled> </snapshots> </reposi tory> <repository> <id>datanucleus</id> <url>http://www.datanucleu                S.org/downloads/maven2</url> <name>Datanucleus</name> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories&gt ;

　

When you're done, continue compiling.

[INFO] Apache Oozie Docs ...... ..... ................ FAILURE [9.477s]

Ping address seems to be possible,

Really do not know why, try a few times not, estimated where needs to change, I have no way, the waiting to solve

2. Change the program, with other compiled good,

So it's Cloudera company.

http://archive.cloudera.com/cdh5/cdh/5/

This: http://archive.cloudera.com/cdh5/cdh/5/oozie-4.1.0-cdh5.8.2.tar.gz

, unzip after download: TAR-ZXVF oozie-4.1.0-cdh5.8.2.tar.gz

This is a hadoop2.6 of support.

Then I'll replace him with my Hadoop version hadoop2.5.2

The Concrete Way is: (thank him)

This reference this address: http://www.mamicode.com/info-detail-490284.html

1. Unzip

CP Oozie-4.1.0-distro.tar.gz/home/hadoop

Cd/home/hadoop

Tar xvzf oozie-4.1.0-distro.tar.gz

/home/hadoop/oozie-4.1.0 is the root directory of the Oozie

2. Setting Environment variables

Vi/etc/profile

Export Oozie_home=/home/hadoop/oozie-4.1.0export path= $PATH: $OOZIE _home/bin

Since then, in the source/etc/profile brought him into force

3. Introduction of the JAR package

Create a Libext folder under Oozie_home

mkdir Libext

Copy all the jar packages for Hadoop into this directory

CP $HADOOP _home/share/hadoop/*/hadoop-*.jar./libext/

CP $HADOOP _home/share/hadoop/*/lib/*.jar./libext/

CP Mysql-connector-java-5.1.29-bin.jar./libext/

Delete Libext in Jasper*.jar, Servlet-api.jar, Jsp-api.jar, and oozie-4.0.1/oozie-server/lib/under the jar package conflict, the war packet will be reported:

Org.eclipse.jdt.internal.compiler.CompilationResult.getProblems () [Lorg/eclipse/jdt/core/compiler/iproblem

4. Create a War package

Bin/oozie-setup.sh Prepare-war

Will generate @oozie_home/oozie-server/webapps/oozie.war

After extracting the Ext-2.2.zip, generate the ext-2.2 folder and enter the folder into Oozie.war. His approach is to start the service after the Oozie.war package will be decompressed to Oozie, so that after the ext-2.2 directly dragged in.

(My practice is, the above Oozie.war, download to the desktop, with the decompression tool to open, and then drag Ext-2.2.zip to Oozie.war, and later found that, in fact, it is not necessary, I opened it)

Note: 1. See the following command on the web to generate Oozie.war, and have ext-2.2.zip into the war package

./addtowar.sh-inputwar $OOZIE _home/oozie.war-outputwar $OOZIE _home/oozie-server/webapps/oozie.war-hadoop 2.3.0 $ Hadoop_home-extjs/home/oozie/ext-2.2.zip

2. You have the zip and unzip command, otherwise it will be packet error, go to the root user, with yum-y install unzip and yum-y installed Zip installation can be

5. Modify the configuration

VI $OOZIE _home/conf/oozie-site.xml

<property>   <name>oozie.service.JPAService.jdbc.driver</name>   <value> com.mysql.jdbc.driver</value>    <description>        jdbc Driver class.    </description></property><property>   <name>oozie.service.jpaservice.jdbc.url</ name>   <value>jdbc:mysql://mysql-server:3306/oozie</value>    <description>        JDBC URL.    </description></property><property>   <name>oozie.service.jpaservice.jdbc.username </name>    <value>root</value>    <description>        DB user name.    </description></property><property>   <name>oozie.service.jpaservice.jdbc.password </name>    <value>mapengbo</value>    <description>        DB user password.    </description></property>

　　

6. Create a database

Create a database named Oozie and assign the right

CREATE DATABASE Oozie;

Grant all on oozie.* to ' shirdrn ' @ ' oozie-server ' identified by ' 0o21e ';

FLUSH privileges;

Generate the required database tables and execute the

bin/ooziedb.sh Create-sqlfile Oozie.sql–run

View database Oozie generated a related table for Oozie.

7. Start the service

bin/oozied.sh start

Accessing the console Http://hadoop1:11000/oozie HADOOP1 for my host name

Four Configure Jobhistory and users for Hadoop

Modify $hadoop_home/etc/hadoop/mapred-site.xml

and $oozie_home/conf/hadoop-conf/core-site.xml Add the following configuration.

<property>       <name>mapreduce.jobhistory.address</name>        <value>node3:10020</ value>     </property>    <property>       <name>mapreduce.jobhistory.webapp.address</ name>        <value>node3:19888</value>     </property>     <property>       <name >mapreduce.jobhistory.intermediate-done-dir</name>       <value>${hadoop.tmp.dir}/mr/history-tmp </value>     </property>    <property>        <name>mapreduce.jobhistory.done-dir< /name>       <value>${hadoop.tmp.dir}/mr/history-done</value></property>

　　

You need to add the following to the Hadoop core-site.xml:

<property>                <name>hadoop.proxyuser.root.hosts</name>                <value>*</value>         </property>         <property>                <name>hadoop.proxyuser.root.groups</name>                < Value>*</value>         </property>

　　

Root for Hadoop user, hadoop.proxyuser.root.groups property Configure user-owned group name, configure complete restart Hadoop

You can also write this: Hadoop.proxyuser. [User].hosts and Hadoop.proxyuser. [User].groups

　

Start the Hadoop history jobhistory Service

$HADOOP _home/sbin/mr-jobhistory-daemon.shstart Historyserver//This is a reboot of the HADOOP cluster

Restart Oozie

bin/oozied.sh start

Five Client Test

TAR–ZXVF oozie-client-4.1.0.tar.gz//This I used to be compiled in the oozie4.2.0 compiled well, found that I downloaded the Cloudera there is no this

Address: Link: http://pan.baidu.com/s/1eSBOdEi Password: Q1NW

TAR–ZXVF oozie-examples.tar.gz

TAR–ZXVF oozie-sharelib-4.1.0.tar.gz

HDSF Dfs-put Examples hdfs:/myserver/user/hadoop/

HDSF dfs-put share/user/hadoop/--//This later found no, need to configure the local directory path in Oozie-site.xml,

Configure Oozie.service.WorkflowAppService.system.libpath

A. Modify the $oozie_home/conf/oozie-site.xml file and add the following:

<property><name>oozie.service.workflowappservice.system.libpath</name><value>file:/// Home/${user.name}/oozie-4.1.0-cdh5.8.2/share/lib</value></property>

B. Modify the $oozie_home/conf/hadoop-conf/core-site.xml file and add the following:

<property>   <name>yarn.resourcemanager.address</name>      <value>node1:8032</ Value> (should be the same as the configuration of Hadoop,                            This I am in http://Your mapreduce hostname £ 8088/conf, and change it to the same) </property><property>     <name>yarn.resourcemanager.scheduler.address</name>      <value>node1:8030</value> </property>

　　

C. Modify the Oozie.service.HadoopAccessorService.hadoop.configurations property to adjust its value to *=hadoop_home/etc/hadoop

---//This I do not have how to configure, you can look at this person configured http://heylinux.com/archives/2836.html

D. Modify $oozie_home/examples/apps/map-reduce/ Job.properties (There is no jobtracker in yarn, the following jobtracker fills in the yarn.resourcemanager.address value, Oozie.wf.application.path is the Oozie sample program in HDFs Path

    namenode=hdfs://node1:9000    jobtracker=node1:8032    queuename=default    examplesroot=examples    Oozie.wf.application.path=${namenode}/user/${user.name}/${examplesroot}/apps/map-reduce    outputDir= Map-reduce

　　

Invoking the OOZIE script in $oozie_home/oozie-client-4.0.1/bin, executing the workflow

./oozie Job-oozie Http://node3:11000/oozie-config $OOZIE _home/examples/apps/map-reduce/job.properties-run

Access console Http://hadoop1:11000/oozie

Completed!

Configure oozie4.10+hadoop2.5.2