Mesos + Docker tutorial:how to Build Your Own Fra

Introduction to Mesos + Docker

Apache Mesos is a cluster manager this simplifies the complexity of running tasks on a shared pool of servers. Docker is a lightweight container for deploying packaged services, similar on concept to a vsan, but without th E overhead.

Mesos added support for Docker in the 0.20.0 release and subsequently fixed some fairly large limitations in the following 0.20.1 patch release. The combination of Mesos + Docker provides a very powerful platform for deploying applications and services in a Clustered environment.

This tutorial would explain how to use Mesos 0.20.1 and Docker 1.2.0 to write a simple Mesos framework in Java that would St Art some containers.

Mesos Overview

A Typical Mesos deployment consists of one or more servers running the Mesos-master (one live instance and one or more STA Ndby instances) and a cluster of servers running the Mesos-slave component. The slaves register with the master and an offer of "resources" i.e. capacity to is able to run tasks. The master then interacts with the deployed frameworks-pass those resource offers on and to receive instructions to run Tasks and then delegates those instructions back to the slaves.

Multiple frameworks can is deployed concurrently and share the resources available in the cluster. For example, Apache Spark and Cassandra both has Mesos frameworks available, allowing them both to being deployed on the SAM E cluster.

A framework consists of a scheduler and optionally one or more executors. The scheduler connects to the Mesos-master and accepts or rejects resource offers from slaves and then provides Instructio NS on the What the tasks to execute on those slaves.

Mesos have default executors for running shell scripts and, since the 0.20.0 release, for launching Docker containers. It is also possible to write executors in Java and other languages. In this case, the executor binary (jar files in the case of Java) must is available from a central resource such as HDFS s o that the slaves can download them. Of course, with the introduction of Docker, there are now the possibility of packaging up the executors directly inside a D Ocker image, making the deployment process much simpler.

Writing a Mesos Framework

This tutorial would demonstrate how to develop a framework with a scheduler that would start Docker containers on one or mor E Slaves. There is no need to develop a executor in this case since the default Mesos executor would be used.

The full source code was available in Github:https://github.com/codefutures/mesos-docker-tutorial

The main class to implement is Examplescheduler which would implement the Scheduler interface.

For this tutorial, only the following methods would be implemented:

void Resourceoffers (Org.apache.mesos.SchedulerDriver schedulerdriver, java.util.List List); void Statusupdate (Org.apache.mesos.SchedulerDriver schedulerdriver, Org.apache.mesos.Protos.TaskStatus taskstatus) ;

The resourceoffers () method would be called whenever there is slaves running with available resource (capacity to Run jobs). The scheduler can then decide whether to accept any of the these offers and schedule any tasks to be run.

The statusupdate () method would be a called to notify the scheduler of the status of the of the the the the tasks that were scheduled. The example code would look in the task status (task_running, task_failed, task_finished) to keep track of how many contain ERs are running.

In this tutorial the scheduler would attempt to maintain a certain number of running tasks. The following instance variables is used to keep track of pending task IDs and running task IDs.

/** List of pending tasks. *  /private final List pendinginstances = new arraylist<> ();  /** List of running tasks. *  /private final List runninginstances = new arraylist<> ();

An atomiclong is used generate sequential task IDs:

/** Task ID Generator. *  /private final Atomicinteger taskidgenerator = new Atomicinteger ();

The main flow in the Resourceoffers () method was to iterate through the list of offers received and decide whether to launch a NY tasks, so the main flow looks like this:

@Override public void Resourceoffers (Schedulerdriver schedulerdriver, List offers) {Logger.info ("resourceoffers () Wit    H {} offers ", Offers.size ());      for (Protos.offer offer:offers) {List tasks = new arraylist<> ();        if (runninginstances.size () + pendinginstances.size () < desiredinstances) {//Generate a unique task ID Protos.taskid TaskID = Protos.TaskID.newBuilder (). SetValue (Integer.tostring (Taskidgenerator.incrementandget ())        ). build ();        Logger.info ("Launching task {}", Taskid.getvalue ());        Pendinginstances.add (Taskid.getvalue ()); Docker Image info Protos.ContainerInfo.DockerInfo.Builder dockerinfobuilder = Protos.ContainerInfo.DockerInfo.ne        Wbuilder ();        Dockerinfobuilder.setimage (ImageName);        Dockerinfobuilder.setnetwork (Protos.ContainerInfo.DockerInfo.Network.BRIDGE);        Container info Protos.ContainerInfo.Builder containerinfobuilder = Protos.ContainerInfo.newBuilder ();Containerinfobuilder.settype (Protos.ContainerInfo.Type.DOCKER);        Containerinfobuilder.setdocker (Dockerinfobuilder.build ()); Create task to run Protos.taskinfo task = Protos.TaskInfo.newBuilder (). SetName ("task" + Taskid.getv Alue ()). Settaskid (TaskId). Setslaveid (Offer.getslaveid ()). Addresources (Protos.resource. Newbuilder (). SetName ("CPUs"). SetType (Protos.Value.Type.SCALAR). Setscalar (P Rotos. Value.Scalar.newBuilder (). SetValue (1)). Addresources (Protos.Resource.newBuilder (). SetName ("Mem "). SetType (Protos.Value.Type.SCALAR). Setscalar (Protos.Value.Scalar.newBuilder (). SetValue (1 )). Setcontainer (Containerinfobuilder). SetCommand (Protos.CommandInfo.newBuilder (). Setshell (False        ). build ();      Tasks.add (Task); } protos.filters Filters = Protos.Filters.newBuilder (). setrefusesecOnds (1). build ();    Schedulerdriver.launchtasks (Offer.getid (), Tasks, filters); }  }

As can see, the This method builds up a task definition. Let's walk through this on more detail.

Since The scheduler is running Docker tasks, the first step was to define the Docker image to be used. The Docker image name must be specified, and optional configuration items include network configuration and port mappings. In this example, bridge networking are used, so that each container gets its own IP address, which is the default Behaviou R when using Docker.

Protos.ContainerInfo.DockerInfo.Builder Dockerinfobuilder = Protos.ContainerInfo.DockerInfo.newBuilder (); Dockerinfobuilder.setimage (imageName);d ockerinfobuilder.setnetwork ( Protos.ContainerInfo.DockerInfo.Network.BRIDGE);

Next, the container information must is specified, mainly providing a reference to the Docker image definition.

Protos.ContainerInfo.Builder Containerinfobuilder = Protos.ContainerInfo.newBuilder (); Containerinfobuilder.settype (Protos.ContainerInfo.Type.DOCKER); Containerinfobuilder.setdocker ( Dockerinfobuilder.build ());

Finally, the task must be defined.

//create task to run protos.taskinfo task = Protos.TaskInfo.newBuil Der (). SetName ("task" + Taskid.getvalue ()). Settaskid (TaskId). Setslaveid (Offer.getslave Id ()). Addresources (Protos.Resource.newBuilder (). SetName ("CPUs"). SetType (Protos . Value.Type.SCALAR). Setscalar (Protos.Value.Scalar.newBuilder (). SetValue (1)). Addresources (Proto S.resource.newbuilder (). SetName ("mem"). SetType (Protos.Value.Type.SCALAR). s Etscalar (Protos.Value.Scalar.newBuilder (). SetValue ()). Setcontainer (Containerinfobuilder). Setco Mmand (Protos.CommandInfo.newBuilder (). Setshell (False)). Build (); 

The task definition specifies the amount of resource needed (1 CPU and MB RAM) and also specifies a reference to the D Ocker container information and the command to run (the "the command is effectively set to NULL, so" the Defau Lt Docker Image entry point would be used).

. Setcontainer (Containerinfobuilder). SetCommand (Protos.CommandInfo.newBuilder (). Setshell (False))

The Statusupdate () method would simply update the pendinginstances and Runninginstances lists based on the task status.

 @Override public void Statusupdate (Schedulerdriver driver,    Protos.taskstatus TaskStatus) {final String taskId = Taskstatus.gettaskid (). GetValue ();    Logger.info ("statusupdate () Task {} is in state {}", TaskId, Taskstatus.getstate ());        Switch (taskstatus.getstate ()) {case TASK_RUNNING:pendingInstances.remove (taskId);        Runninginstances.add (TASKID);      Break        Case Task_failed:case TASK_FINISHED:pendingInstances.remove (taskId);        Runninginstances.remove (TASKID);    Break  } logger.info ("Number of instances:pending={}, running={}", Pendinginstances.size (), runninginstances.size ()); }

The scheduler'll be created by the framework, which are basically just the main () method that's invoked from the command Line. The framework creates the scheduler and registers it with Mesos:

Frameworkinfo.builder Frameworkbuilder = Frameworkinfo.newbuilder ()        . SetName ("Codefuturesexampleframework")        . SetUser ("")//Mesos fill in the current User:        Setfailovertimeout (frameworkfailovertimeout); Timeout in seconds    final Scheduler Scheduler = new Examplescheduler (        imageName,        totaltasks    );    Mesosschedulerdriver Driver = new Mesosschedulerdriver (Scheduler,     frameworkbuilder.build (), masteripandport);    Driver.run ();

Running the Tutorial Code

These instructions is for running everything in a single node, but can easily is adapted to run on multiple nodes.

Running Mesos Master and Slave:

Nohup mesos-master--ip=127.0.0.1--work_dir=/tmp >mesos-master.log 2>&1 &nohup mesos-slave--master= 127.0.0.1:5050--containerizers=docker,mesos >mesos-slave.log 2>&1 &

Checkout The code () from the GitHub repo:

git clone Https://github.com/codefutures/mesos-docker-tutorial.gitcd MESOS-DOCKER-TUTORIALMVN Package

Run the code:

Launch the framework to run 2 instances of the Fedora/apache image.

Java-classpath Target/cf-tutorial-mesos-docker-1.0-snapshot-jar-with-dependencies.jar Com.codefutures.tutorial.mesos.docker.ExampleFramework 127.0.0.1:5050 Fedora/apache 2

The framework should output logging like this:

Running ' Docker ps ' should confirm that the containers has been launched:

This Mesos +docker tutorial should provide you the tools to do the things:

1. Write a Mesos Framework
2. Use the new Docker support in Mesos

Get The full source code, available in Github:https://github.com/codefutures/mesos-docker-tutorial

Image by Damien Gabrielson

Mesos + Docker tutorial:how to Build Your Own Fra