Implementation of Docker on yarn in Hulu

This article is the main work I have done in Hulu this year, combined with the current popular two open source solutions Docker and yarn, provide a flexible programming model, currently supporting the DAG programming model, will support the long service programming model.

Based on Voidbox, developers can easily write a distributed framework, Docker as a running execution engine, yarn as a management system for cluster resources.

This article is also published on Hulu's Official technology blog: http://tech.hulu.com/blog/2015/08/06/voidbox-docker-on-yarn/

1. Voidbox Motivation

YARN is the distributed resource Management system in Hadoop 2.0, which are able to schedule cluster resources for diverse High-level applications such as MapReduce, Spark. However, nowadays, all existing the framework on top of YARN is designed with assumption of specific system environment. How to support user applications with arbitrary complex environment dependencies are still an open question. Docker gives the answer.

Docker is a very popular container virtualization technology. It provides a-to-run almost any application isolated in a container. Docker is a open platform for developing, shipping, and running applications. Docker automates the deployment of any application as a lightweight, portable, self-sufficient container that'll run Vir Tually anywhere.

in order to integrate the unique advantages of Docker and YARN, the Hulu engineering team developed Voidbox. Voidbox enables any application encapsulated in Docker image running on YARN cluster along with MapReduce and Spark. Voidbox brings the following benefits:

• Ease Creating Distributed Application
  • Voidbox handles most common issues in distributed computation system, say it, cluster discovery, elastic resource Allocati On, task coordination, disaster recovery. With its well-designed interface, it's easy to implement a distributed application.
• Simplify deployment
  • Without voidbox, we need to create and maintain dedicated VMS for application with complex environment even though the VM I Mage is huge and isn't easy to deploy. With Voidbox, we could easily get resource allocated and make app run right the time we need it. Additional maintenance work is eliminated.
• Improve cluster efficiency
  • As we could deploy SPARK/MR and all kinds of voidbox applications from different department together, we could maximize CL Uster usage.

Thus, YARN as a big data operating platform has been further consolidated and enhanced.

Voidbox supports Docker container-based DAG (Directed acyclic Graph) tasks in execution. Moreover, Voidbox provides several ways to submit applications considering demands of the production environment and the D ebugging environment. In addition, Voidbox can cooperate with Jenkins, GitLab and private Docker Registry to set up a set of developing, testing , automatic release process.

2.Voidbox Architecture 2.1 YARN Architecture Overview

YARN enables multiple applications to share resources dynamically in a cluster. Here are the architecture of applications running in YARN cluster:

Figure 1. YARN Architecture

As shown in Figure 1, a client submits a job to Resource Manager. The Resource Manager performs its scheduling function according to the Resource requirements of the application. Application Master is responsible for the application tasks scheduling and execution of a application ' s lifecycle.

Functionality of each modules:

• Resource manager:responsible for Resource management and scheduling in cluster.
• Nodemanager:running on the compute nodes in cluster, taking care of tasks execution in the individual machine, collecting Informations and keeping Heartbeat with Resource Manager.
• Application Master:takes Care of requesting resources from YARN, then allocates resources to run tasks in Container.
• Container:container is an abstract notion which incorporates elements such as memory, CPU, disk, network etc.
• hdfs:distributed file system in YARN cluster.

2.2 Voidbox Architecture Design

In Voidbox architecture, YARN was responsible for the cluster ' s resource management. Docker acts as the task execution engine above of the operating system, cooperating with Docker Registry. Voidbox helps to translate user programming code into Docker container-based DAG tasks, apply for resources according to R Equirements and deal with dags in execution.

Figure 2. Voidbox Architecture

As shown in Figure 2, each of the box stands for one machine with several modules running inside. The architecture more clearly, we divide them into three parts, and functionality of Voidbox modules and Docker mo Dules:

• Voidbox Modules:
  • Voidbox client:the Client Program. Through Voidbox Client, users can submit a Voidbox application, stop it, and so on. By the To, Voidbox application contains several Docker jobs and a docker job contains one or more Docker tasks.
  • Voidbox master:actually, it's a application Master in yarn, and takes care of the requesting resources from yarn, then alloc Ates resources to Docker tasks.
  • Voidbox driver:responsible for task scheduling of a single voidbox application. Voidbox supports Docker container-based DAG task scheduling and between tasks we can insert some other codes. So Voidbox Driver should handle the order scheduling of the DAG task dependencies and execute the user s code.
  • Voidbox Proxy:the bridge between yarn and Docker engine, responsible for transiting commands from YARN to Docker engine, such as Start or kill Docker container, etc.
  • State server:maintaining The informations of Docker engine ' s health status, providing the list of machines which can run Docker container. So Voidbox Master can apply to resources more efficiently.
• Docker Modules:
  • Docker Registry:docker Image storage, acting as an internal version control tool of Docker image.
  • Docker engine:docker Container execution Engine, obtaining specified Docker image from Docker Registry and launching Dock ER container.
  • Jenkins:cooperating with GitLab, when application codes update, Jenkins would take care of automated testing, packaging, G Enerating the Docker image and uploading to Docker Registry, to complete the application automatically release process.

2.3 Running Mode

Voidbox provides, application running Modes:yarn-cluster mode and yarn-client mode.

In Yarn-cluster mode, the control component and resource management component is running in the yarn cluster. After we submit the Voidbox application, Voidbox Client can quit at any time without affecting the running time of Applica tion. It ' s for the production environment.

In yarn-client mode, the control component was running in Voidbox client, and other components of the cluster. Users can see much more detailed logs about the application ' s status. When Voidbox Client quits, the application in cluster would exit too. So it's more convenient for debugging.

Here we briefly introduce the implementation architecture of the of the modes:

• Yarn-cluster mode

Figure 3. Yarn-cluster mode

As shown in Figure 3, Voidbox Master and Voidbox Driver is both running in the cluster. Voidbox Driver is responsible for controlling the logic and Voidbox Master takes care of application resource management.

• Yarn-client mode

Figure 4. Yarn-client mode

As shown in Figure 4, Voidbox Master are running in the cluster, and Voidbox Driver are running in Voidbox Client. Users can submit Voidbox application in IDE for debugging.

2.4 Running Procedure

Here is the procedures of submitting a Voidbox application and its lifecycle:

2. Users Write a Voidbox application by Voidbox SDK and generate a Java archive, then submit it to the YARN cluster by voidbo x Client;
4. After receiving Voidbox application, Resource Manager would allocate resources for Voidbox Master, then launch it.
6. Voidbox Master starts Voidbox Driver, the latter would decompose Voidbox application into several Docker jobs (a job contain s one or more Docker tasks). Voidbox Driver calls Voidbox Master interface to launch the Docker tasks in compute nodes.
8. Voidbox Master requests resources from Resource Manager, and Resource manager allocates some YARN containers according to The YARN cluster status. Voidbox Master launches Voidbox Proxy in YARN container, and the latter are responsible for communication with Docker Engin E to start the Docker container.
10. User ' s Docker task is running in Docker container, and the log output to a local file. User can see real-time application logs through YARN Web Portal.
12. After all Docker tasks is done, the logs is aggregate to HDFS, so user still can get the application logs by history Server.

2.5 Docker integrating with YARN in resource management

YARN acts as a Uniform Resource manager in the cluster, and are responsible for resource management on all machines. Docker as a container engine also has the function of resource management. Integrate their resource management function is particularly important.

In yarn, the user task can is only run in the YARN container, while Docker container can is only is handled by Docker engine. This case would get out of the management of yarn and damage the unified management and scheduling principle of yarn, whic H could produce resource leaks risk issue. In order to enable yarn to manage and schedule Docker container, we need to build a proxy layer between YARN and Docker en Gine. This is what Docker Proxy is introduced. Through Voidbox Proxy, YARN can manage the container lifecycle including start, stop, etc.

In order to understand voidbox proxies more clearly, we take stopping voidbox application as an example. When a user needs to kill Voidbox application, YARN would recycle all the resources of the application. At this point, YARN would send a kill signal to the related machines. The corresponding Voidbox Proxy would catch the kill signal, then stop Docker container in Docker engine to do the resource Recycling. So with the help of Voidbox proxies, it can not be only stop YARN container, but also stop the Docker container to avoid resour Ces leaks issue (this was the problem existing in open source version, see YARN-1964).

3. Fault tolerance

Although Docker has some stable releases, the enterprise production environment have a variety versions of operating system or kernel, so it brings unstable factors. We consider multiple levels in Voidbox fault-tolerant design to ensure Voidbox ' s high availability.

• Voidbox Master Fault Tolerance
  • If Resource Manager finds Voidbox Master crashes, it'll notify NodeManager to recycle all the YARN containers belonging To the Voidbox application, then restart Voidbox Master.
• Voidbox Proxy Fault Tolerance
  • If Voidbox Master finds Voidbox proxy crashes, it would recycle Docker containers on behalf of Voidbox Proxy.
• Docker Container Fault Tolerance
  • Each Voidbox application can configure the maximum retry times on failure when the Docker container crashes, Voidbox Mast Er would do some work according to the exit code of Docker container.

4. Programming model 4.1 DAG programming Model

Voidbox provides Docker container-based DAG programming model. A sample would look similar to this:

Figure 5. Docker container-based DAG programming model

As shown in Figure 5, there is four jobs on this Voidbox application, and the job can configure its requirements of CPUs, Memory, Docker image, parallelism and so on. JOB3 would start when Job1 and Job2 both complete. Job1, JOB2 and Job3 make a stage, so user can inserts their codes after the this stage was done, and finally start running JOB4.

4.2 Shell mode to submit one task

In the most cases, we would like to run a single Docker container-based task without programming. So Voidbox supports Shell mode to describe and submit the Docker container-based task, actually it ' s a implementation base D on DAG programming mode. The example usage of Voidbox in shell mode:

docker-submit.sh \

-docker_image CentOS \

-shell_command "echo Hello voidbox" \

-container_memory 1000 \

-cpu_shares 2

The shell script above would submit a task to run "Echo Hello Voidbox" in a docker image named ' CentOS ', and the resource R Equirement is 1000Mb memory, 2 CPU virtual cores.  

5. Voidbox in Action

At present we can run Docker, MapReduce, Spark and other applications in YARN cluster. There has been lots of the short tasks using Voidbox within HULU.

• Automation Testing Process
  • Cooperating with Jenkins, GitLab and private Docker registry, when the application codes Update, Jenkins would complete AUT Omatic test, package program, regenerate Docker image and push it to the private Docker Registry. It's a process of development, testing and automatically release.
• Complex tasks in parallel
  The
  • Test Framework is a used to does some testings to detect the availability of some. The project is implemented by Ruby/java and have complex dependencies. So we maintain the layers of Docker image, the first layer was the system software as a base image, and the second layer is The business level. We Publish a test framework Docker image and use some timing scheduling software to start Voidbox application regularly. Thanks to Voidbox, we solve the issues such as the complex dependencies and the multitasking parallelism.
  • Facematch (link:http://tech.hulu.com/blog/2014/05/03/face-match-system-overview/) is a video analysis Application. It ' s implemented by C and have lots of graphics libraries. That can is optimized by voidbox:first of any we need to package all face match program into a Docker image and then write V Oidbox application to handle the multiple videos. Voidbox solves the complex machine environment and the parallelism control problem.
• Building Complex Workflow
  • Some tasks has a dependent with all other, such as it needs to load user behaviors first, and then does the analysis of User B Ehaviors. These and steps have successively dependencies. We Use the Voidbox container-based programming model to handle the case easily.

6. Different from Dockercontainerexecutor in YARN 2.6.0

• Dockercontainerexecutor (link:https://issues.apache.org/jira/browse/yarn-1964) is released in YARN 2.6.0 and it ' s alpha Version. Not mature enough, and it's only a encapsulation layer above the default executor.
• Dockercontainerexecutor is difficult to coexist with other containerexecutor in one YARN cluster.
• Voidbox Features
  • DAG programming Model
  • Configurable container level of fault tolerance
  • A variety of running modes, considering development environment and production environment
  • Share YARN cluster resources with other Hadoop job
  • Graphical Log View tool

7. Future work

• Support more versions of YARN
  • Voidbox would like-to-support + versions in the future besides YARN 2.6.0.
• Voidbox Master fault tolerance, persistent metadata to reduce the cost of a case of retry
  • Currently, if a voidbox Master crashes, YARN would recycle resources belonging to this Voidbox application and restart Void Box Master to does some tasks from the very beginning. It ' s not necessary to impact tasks which is already done or running. We might keep some metadatas in the state Server to reduce the cost in case of Voidbox Master on-failure.
• Voidbox Master as a permanent service
  • Voidbox would support long running Voidbox Master to receive streaming tasks.
• Support Long Service
  • Voidbox would support long running service if Voidbox Master ' s downtime doesn ' t influence running task.

Implementation of Docker on yarn in Hulu