Set up Azkaban 2.5.0

I. Preface

I recently tried to build an Azkaban by referring to the official document. I found that there are many pitfalls in this document, so I will record it here.

II. Environment and software

Installation environment:

• System Environment: ubuntu-12.04.2-server-amd64
• Installation Directory:/usr/local/AE/ankaban
• JDK installation directory: Export java_home =/usr/local/AE/jdk1.7.0 _ 51
• Hadoop installation directory export hadoop_home =/usr/local/AE/hadoop-1.2.1
• MySQL version: mysql-server-5.5

Software required:

• Azkaban-web-server-2.5.0.tar.gz
• Azkaban-executor-server-2.5.0.tar.gz
• Azkaban-sql-script-2.5.0.tar.gz
• Azkaban-jobtype-2.5.0.tar.gz

Azkaban Source: github.com/azkaban/azkaban

Azkaban plugin Source: github.com/azkaban/azkaban-plugins

DOC: Azkaban. GitHub. IO/Azkaban/docs/2.5/

3. Configure MySQL

1. Decompress azkaban-sql-script-2.5.0.tar.gz
   

   [email protected]:/usr/local/ae/azkaban$ tar -zxvx azkaban-sql-script-2.5.0.tar.gz

2. Log on to MySQL to create database Azkaban.
   

   [email protected]:/usr/local/ae/azkaban$ mysql -u root -pEnter password:Welcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 165Server version: 5.5.37-0ubuntu0.12.04.1 (Ubuntu)Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.mysql> create database azkaban;

3. Create an Azkaban table
   

   mysql> use azkabanReading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql> source /usr/local/ae/azkaban/azkaban-2.5.0/create-all-sql-2.5.0.sql

4. Create a user named Azkaban for Azkaban
   

   mysql> grant all privileges on azkaban.* to ‘azkaban‘@‘localhost‘ identified by ‘azkaban‘;mysql> flush privileges;

4. Configure Azkaban-Web

1. Extract azkaban-web-server-2.5.0.tar.gz
   

   [email protected]:/usr/local/ae/azkaban$ tar -zxvx azkaban-web-server-2.5.0.tar.gz

2. Generate an SSL Certificate
   For details about how to use Java keytool to generate keystore and truststore files, refer to my previous articles.
   Here you can simply generate the keystore file and copy the generated keystore file to the/usr/local/AE/Azkaban/azkaban-web-2.5.0/Web file.
   In this document, the certificate file is keystone and the keypass is kestore.
3. Modify./CONF/Azkaban. Properties
   

   #Azkaban Personalization Settingsazkaban.name=Azkabanazkaban.label=My Local Azkabanazkaban.color=#FF3601azkaban.default.servlet.path=/indexweb.resource.dir=../web/default.timezone.id=America/Los_Angeles#Azkaban UserManager classuser.manager.class=azkaban.user.XmlUserManageruser.manager.xml.file=../conf/azkaban-users.xml#Loader for projectsexecutor.global.properties=../conf/global.propertiesazkaban.project.dir=../projectsproject.temp.dir=../temptrigger.plugin.dir=../plugins/triggersdatabase.type=mysqlmysql.port=3306mysql.host=localhostmysql.database=azkabanmysql.user=azkabanmysql.password=azkabanmysql.numconnections=100# Velocity dev modevelocity.dev.mode=false# Azkaban Jetty server properties.jetty.maxThreads=25jetty.ssl.port=8443jetty.port=8081jetty.keystore=../web/keystorejetty.password=keystorejetty.keypassword=jetty-azkabanjetty.truststore=../web/keystorejetty.trustpassword=keystore# Azkaban Executor settingsexecutor.port=12321# mail settingsmail.sender=***********************mail.host=***********************mail.user=************************mail.password=******job.failure.email=*************************job.success.email=*************************lockdown.create.projects=falsecache.directory=../cache
   

   * The options marked in red in the above text are different from those configured on the official website. If some options are selected, an exception or error log is thrown.

4. Start Azkaban-Web
   

   [email protected]:/usr/local/ae/azkaban/azkaban-web-2.5.0/bin$ sh azkaban-web-start.sh

5. Log on to https: ae01: 8443 Username: Azkaban; Password: Azkaban
6. Modify the Azkaban-web Startup File
   If the file cannot be uploaded, You need to modify the startup script azkaban-web-start.sh for Azkaban-Web
   

   if [[ -z "$tmpdir" ]]; then    --->    if [ -z "$tmpdir" ]; then

5. Configure Azkaban-executor

1. Extract azkaban-executor-server-2.5.0.tar.gz
   

   [email protected]:/usr/local/ae/azkaban$ tar -zxvx azkaban-executor-server-2.5.0.tar.gz

2. Configure./CONF/Azkaban. Properties
   

   #Azkabandefault.timezone.id=America/Los_Angeles# Azkaban JobTypes Pluginsazkaban.jobtype.plugin.dir=plugins/jobtypes#Loader for projectsexecutor.global.properties=../conf/global.propertiesazkaban.project.dir=../projectsazkaban.execution.dir=../executionsproject.temp.dir=../tempdatabase.type=mysqlmysql.port=3306mysql.host=localhostmysql.database=azkabanmysql.user=azkabanmysql.password=azkabanmysql.numconnections=100# Azkaban Executor settingsexecutor.maxThreads=50executor.port=12321executor.flow.threads=30

   * The options marked in red in the above text are different from those configured on the official website. If some options are selected, an exception or error log is thrown.

3. Configure the jobtype plug-in
   Unzip the azkaban-jobtype-2.5.0.tar.gz to./plugins and rename it jobtypes
   

   [email protected]:/usr/local/ae/azkaban/azkaban-executor-2.5.0/plugins$ tar -zxvx azkaban-jobtype-2.5.0.tar.gz[email protected]:/usr/local/ae/azkaban/azkaban-executor-2.5.0/plugins$ mv ./azkaban-jobtype-2.5.0 ./jobtypes

   Configure./CONF/common. propertes

   ## everything that the user job can knowhadoop.home=/usr/local/ae/hadoop-1.2.1#hive.home=#pig.home=azkaban.should.proxy=truejobtype.global.classpath=${hadoop.home}/hadoop-core-1.2.1.jar,${hadoop.home}/conf

4. Start Azkaban-executor
   

   [email protected]:/usr/local/ae/azkaban/azkaban-executor-2.5.0/bin$ sh azkaban-executor-start.sh