Azkaban 3.0 Distributed Installation Deployment documentation

Azkaban 3.0 Distributed installation Deployment documentation

A Brief introduction

The Azkaban2.5 supports two modes of operation:

1.solo server Mode:web and executor are executed in the same process for H2 DB.

2.two server Mode:web and executor execute (running on the same server) database for MySQL in different processes.

However, none of the above is applicable to distributed deployments, with 1 web-executors. This condition was fixed at more than 3.0.

AZKABAN3.0 adds new modes of operation on the basis of the original two types:

Multiple executor Mode:web and executor can run on different servers, DB uses MySQL.

Two Environment configuration

Since there is no corresponding installation package above azkaban3.0, it is necessary to compile from the source code. The compiled environment requires the installation of JDK8.

The project is a Gradle project.

Three compilation, installation process

1.git clone Git@github.com:azkaban/azkaban.git. Azkaban source code, currently 3.24 version

2. Enter the Azkaban project to compile./gradlew Disttar

The result of the compilation is:

Azkaban-common: Common Tool class.

AZKABAN-DB: Corresponding SQL script

Azkaban-hadoop-secutity-plugin:hadoop about Kerberos Plugins

Azkaban-solo-server:web and executor projects together.

Azkaban-web/executor-server:azkaban Web and Executor server information

Azkaban-spi:azkaban Storage interface and Exception class

3. Execute SQL under AZKABAN-DB in the database:

Create all table Create-all in the database. After execution, then execute the upgrade.3.20 to 3.22.sql under the source code.

4. After the compilation is successful, azkaban-exec/web-server the following directory to get the tar file

5. Unzip Azkaban-exec Azkaban-web Azkaban-solo

Copy the Conf under the solo to Azkaban-exec, azkaban-web delete the global.properties.

Copy the plugins under the solo to Azkaban-exec, or the following error will occur when the workflow is executed:

This is because the Azkaban needs to verify the submitted user information, will plugins/jobtypes/commonprivate.properties this check off the
6. Create the Logs folder under Azkaban-exec and Azkaban-web and copy the Mysql-connect jar to the Lib folder.

7. Multiple executor only need to be configured to link to the same database. Executor will automatically query executor this table, but the default active is 0, and you need to manually modify active for 1.azkaban-web to register these executor at startup. Note that adding executor requires restarting the Web.

8.executor configuration file Modification/azkaban-web/conf/azkaban.properties: add:

#Multiple Executor Mode

#open it if you use the multiple executor mode

Azkaban.use.multiple.executors=true

#azkaban. Executorselector.filters=staticremainingflowsize,minimumfreememory,cpustatus

Azkaban.executorselector.filters=staticremainingflowsize,cpustatus

Azkaban.executorselector.comparator.numberofassignedflowcomparator=1

Azkaban.executorselector.comparator.memory=1

Azkaban.executorselector.comparator.lastdispatched=1

Azkaban.executorselector.comparator.cpuusage=1

Set the mode of Azkaban to multiple executor. The second layer is to filter some executor distribution conditions.

9. Copy the log4j file to azkaban-exec and Azkaban-web or it will appear at startup

Exception:java.lang.StackOverflowError thrown from the Uncaughtexceptionhandler in thread "main"

10. As of Azkaban Web Jetty keystore configuration, you can use the ... PS. Start executor and start the web first.