Manage Docker clusters with Mesos and marathon

Distributed systems are difficult to understand, design, build, and manage, and they introduce variables that multiply more than a single machine into the design, making it harder to find the root cause of the application. SLAs (Service level agreements) are the standard for measuring downtime and/or performance degradation, and most modern applications have an expected level of resiliency SLA, typically increased by "9" (e.g. 99.9 or 99.99% monthly availability). Each additional 9 becomes more and more difficult to achieve.

Distributed systems are usually partitioned by static partitioning, such as Akka/play, Spark/hadoop, Storm, and Redis. The disadvantage of static partitioning is increased complexity, as the number of machines increases, software management becomes more complex, and failure management is increasingly difficult to maintain. And the resource consumption is very not economical, is the static partition under the resource utilization:

Apache Mesos can run multiple types of distributed systems on the same cluster machine, with more dynamic and inefficient sharing of resources. Provides failure detection, task release, task tracking, task monitoring, low level resource management, and fine-grained resource sharing, scaling up to thousands of nodes. Mesos has been used by Twitter to manage their data centers.

The composition of the Mesos frame is as follows:

The Mesos framework is an application that runs distributed applications on Mesos and has two components:

1. Scheduler: Interacts with Mesos, subscribes to resources, and then loads tasks from the server in Mesos.
   

2. Executor: Obtain information from the framework's environment variable configuration and run the task from the server in Mesos.

Let's see how it implements resource invocation. Mesos resources are allocated through the resources offers, which is actually a snapshot of the currently available resources, which the scheduler uses to run tasks on Mesos from the server.

The sequence diagram of the Mesos master-Slave Server scheduling resource is as follows:

First, the Mesos primary server queries the available resources to the scheduler, the second scheduler sends the load task to the primary server, the primary server communicates to the slave server, loads the task execution from the server to the executor command, the executor executes the task, reports the state feedback to the slave server, and finally informs the scheduler.

Managing multiple actuators from the server, each executor is a container that used to use the Linux container LXC and now uses the Docker container.

Failure recovery and high availability

Mesos Primary server uses zookeeper for service election and discovery. It has a registrar that records all running any and from server information, using Multipaxos for log replication to achieve consistency.

Mesos has a recovery mechanism from the server, no matter when a server freezes, the user's task is able to continue to run, from the server will be some key information such as task information status updates persisted to the local disk, restart can be resumed from disk to run these tasks ( Similar to passivation and wakeup in Java)

What is Marathon

It is a mesos framework that can support running long services, such as Web applications. is a clustered distributed INIT.D, capable of running any Linux binary release as-is, such as Tomcat play, and so on, can be clustered multi-process management. It is also a private pass that enables discovery of services, provides rest API services for deployment, has authorization and SSL, configuration constraints, and enables service discovery and load balancing through Haproxy.

In this way, we can manage thousands of servers like a Linux host, and their corresponding principles, such as the use of marathon similar to the Linux host in the Init SYSTEMD and other shell management, and Mesos is not only a Linux kernel, The Linux kernel, which can dispatch thousands of servers, is actually the core of a data center:

Installation configuration

First, we need to set up the Mesos cluster environment, below which we set Mesos Master/slave and Zookeeper on the Ubuntu 14.04 vagrant node. Installation dependencies:

$ apt-get Install Curl Python-setuptools python-pip Python-dev python-protobuf

Install Zookeeper:

$ apt-get Install Zookeeperd

After installation, Zookeeper has a configuration, and each zookeeper needs to know where it is in quorum.

$ echo 1 | sudo dd Of=/var/lib/zookeeper/myid

Set up Docker below:

$ echo "Deb Http://get.docker.io/ubuntu Docker main" >/etc/apt/sources.list.d/docker.list

$ apt-get Update && apt-get install Lxc-docker

$ docker Version

Client version:1.0.0 Client API version:1.12 Go version (client): go1.2.1 Git commit (client): 63fe64c S erver version:1.0.0 Server API version:1.12 Go version (server): go1.2.1 Git commit (server): 63fe64c

Pull an Ubuntu image from the Docker hub

$ Docker Pull Libmesos/ubuntu

Configuration Mesos:

$ CURL-FL Http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.19.0~ubuntu14.04%2B1_amd64.deb-o/tmp/ Mesos.deb

$ dpkg-i/tmp/mesos.deb

$ mkdir-p/etc/mesos-master

$ echo In_memory | sudo dd of=/etc/mesos-master/registry

# # Mesos Python Egg for use in authoring frameworks

$ CURL-FL Http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.19.0_rc2-py2.7-linux-x86_64.egg-o/tmp/ Mesos.egg

$ easy_install/tmp/mesos.egg

Download Marathon:

$ tar xvzf marathon-0.6.0.tgz

Mesos manages Docker via Deimos and installs Deimos via PIP:

$ pip Install Deimos

Configure Mesos to use Deimos

$ mkdir-p/etc/mesos-slave

$ Echo/usr/local/bin/deimos | sudo dd Of=/etc/mesos-slave/containerizer_path

$ echo External | sudo dd of=/etc/mesos-slave/isolation

Start all services:

$ initctl Reload-configuration

$ service Docker start

$ Service Zookeeper Start

$ service Mesos-master Start

$ service Mesos-slave Start

##### starting Marathon #####

$ CD marathon-0.6.0

$./bin/start--master Zk://localhost:2181/mesos--zk_hosts localhost:2181

Marathon has started listening on port 8080 and we can access it through the browser:

Curl Localhost:8080/help # gives us some details about the API ' s

Start container

Our rest API to launch a container via marathon is submitted as follows:

Curl-x post-h "Accept:application/json"-H "Content-type:application/json" \
Localhost:8080/v2/apps-d ' {
"Container": {"image": "Docker:///libmesos/ubuntu", "Options": ["--privileged"]},
"CPUs": 0.5,
"cmd": "Sleep 500",
"id": "Docker-tester",
"Instances": 1,
"Mem": 300
}‘

We can check the syslog by submitting a custom Docker with the option "options" after the Curl command, because the Mesos default log is placed in the syslog.

June 07:24:58 vagrant-ubuntu-trusty-64 deimos[19227]: deimos.containerizer.docker.launch () exit 0//Docker run--sig-p Roxy--rm--cidfile/tmp/deimos/mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/cid-w/tmp/mesos-sandbox-v/tmp/deimos/ Mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/fs:/tmp/mesos-sandbox--privileged-p 31498:31498-c 512-m 300m-e PORT= 31498-e port0=31498-e ports=31498 libmesos/ubuntu sh-c ' sleep 500 '

We can also check the status of our startup tasks through the marathon Rest API:

Curl-x get-h "Content-type:application/json" Localhost:8080/v2/apps

Here is the marathon UI for information:

More information about Docker tasks can be 5050 through the Mesos GUI on the port of the Mesos master server, now we test the scalability of the task, such as we need to join more node servers, there are two ways to use the GUI or send a put request.

Curl-x put-h "Content-type:application/json" localhost:8080/v2/apps/docker-tester \
"Container": {"image": "Docker:///libmesos/ubuntu", "Options": ["--privileged"]},
"CPUs": 0.5,
"cmd": "Sleep 500",
"id": "Docker-tester",
"Instances": 2, # Increasing the instance count to 2
"Mem": 300
}‘

Marathon ensures that all Docker processes start running, and if a process crashes, marathon restarts the same process to ensure that each configuration runs an instance, as well as other open source Mesos schedulers such as Apache Aurora, Airbnb's Chronos. But Marathon is relatively straightforward to provide a good rest API for managing containers, although Mesos, Marathon and Docker are still young, but provide a combination of Docker-based killer cluster management.

Manage Docker clusters with Mesos and marathon