My experience in running Docker on Mesos

My experience in running Docker on Mesos

My technology Stack

As I wrote in this blog, I want to share with you how to use docker containers in a single cluster to run analysis jobs. In Zions, we are studying Apache Mesos running on the MapR platform and MapR-FS. My goal is to try to build a ubiquitous computing platform. For analysis, I run Spark and Myriad (more development is required for MapR and other platforms in this field ). Myriad is used to run MapReduce jobs. I use Kafka and Storm running on mesos at the same time, used together with the MapR file system, or used together with the current environment.

MapR provides great help when it runs Docker containers on mesos. One example is the Hive metastore service that I run inside docker. Because Hive metastore requires a relational database to persist table metadata, I need to deploy a MySQL service instance at the same time. I publish this instance in a Mesos-based Docker using Marathon instead of deploying MySQL on an independent server outside the cluster. Because the data stored in MySQL is very important, I want to confirm that if the container crashes or its host machine dies, Marathon can create a new container and take over all of its offline intact data. The NFS function of MapR-FS is very simple to implement because of its random read/write capability and high performance to maintain load for a database.

Use the MapR File System

One problem that needs to be solved is that when a MySQL database is enabled, it requires exclusive access to database files. I want to prevent another unexpected Docker container from running on these files, this is because access to the same file from MySQL with two instances is not a good thing if you want to maintain consistency of your database files. So I have a deep understanding of it with Ted Dunning and Keys Botzum in MapR, and I ask them, "How do I use a lock ?" Although MapR NFS does not support locking from a traditional Unix perspective, MapR supports enabling locking by creating directories and file systems that create new files.

Using their suggestions, I wrote a script to implement the locking mode, which allows reliable persistent data storage. This means that others can also benefit from it, so I will share this script here.

There are two parts. The first part is, "I want to use a lock on the file and I want it to be exclusive ". This is not supported, but MapR also supports creating a directory in semantics and the only possible way to create this directory, which is also used in this script, I want to create something my Docker container can find and say, "Some people use this data at the same time and I need to disable it." My script prevents MySQL or Hive Metastore running on two different instances on my cluster, but I still have the ability to run MySQL on any node of my cluster.

There is no limit on where it runs. One way the Mesos community tries to solve this problem is to persist data to different frameworks-so you can use this data block-and it will be included in future versions. However, MapR has a high-performance file system available on all my nodes, so I want to make better use of it.

My code written for the Docker container to process the file system lock

Basically, this code runs like a gasket. I call this code instead of enabling any process that I expect to enable directly in Mesos. This Code detects each specific directory that I set. For example, if it is a MySQL or Minecraft Docker container, it will detect a separate address for the container. My Minecraft server has an address in MapR-FS; this is exactly how it detects and decides if it can hold an exclusive lock on this directory and run it. If it cannot do this-it finds that some processes also lock the directory, it knows that it cannot run and turn off the container. This ensures that I will not have more than one Docker container of the same type. I don't want two Minecraft servers to run because they will work on the same data, resulting in file corruption.

Here is the code I wrote for The Docker container to process the file system lock:

#! /Bin/bash

# The location the lock will be attempted in

LOCKROOT = "/minecraft/lock"

LOCKDIRNAME = "lock"

LOCKFILENAME = "mylock. lck"

# This is the command to run if we get the lock.

RUNCMD = "./start. sh"

# Number of seconds to consider the Lock stale, this cocould be application dependent.

LOCKTIMEOUT = 60

SLEEPLOOP = 30

LOCKDIR =$ {LOCKROOT}/$ {LOCKDIRNAME}

LOCKFILE =$ {LOCKDIR}/$ {LOCKFILENAME}

If mkdir "$ {LOCKDIR}" &>/dev/null; then

Echo "No Lockdir. Our lock"

# This means we created the dir!

# The lock is ours

# Run a sleep loop that puts the file in the directory

While true; do date + % s> $ LOCKFILE; sleep $ SLEEPLOOP; done &

# Now run the real shell scrip

$ RUNCMD

Else

# Pause to allow another lock to start

Sleep 1

If [-e "$ LOCKFILE"]; then

Echo "lock dir and lock file Checking Stats"

CURTIME =date +%s
FILETIME =cat $LOCKFILE

DIFFTIME = $ ($ CURTIME-$ FILETIME ))

Echo "Filetime $ FILETIME"

Echo "Curtime $ CURTIME"

Echo "Difftime $ DIFFTIME"

If ["$ DIFFTIME"-gt "$ LOCKTIMEOUT"]; then

Echo "Time is greater then Timeout We are taking Lock"

# We shoshould take the lock! First we remove the current directory because we want to be atomic

Rm-rf $ LOCKDIR

If mkdir "$ {LOCKDIR}" &>/dev/null; then

While true; do date + % s> $ LOCKFILE; sleep $ SLEEPLOOP; done &

$ RUNCMD

Else

Echo "Cannot Establish Lock file"

Exit 1

Fi

Else

# The lock is not ours.

Echo "Cannot Estblish Lock file-Active"

Exit 1

Fi

Else

# We get to be the locker. However, we need to delete the directory and recreate so we can be all atomic about

Rm-rf $ LOCKDIR

If mkdir "$ {LOCKDIR}" &>/dev/null; then

While true; do date + % s> $ LOCKFILE; sleep $ SLEEPLOOP; done &

$ RUNCMD

Else

Echo "Cannot Establish Lock file-Issue"

Exit 1

Fi

Fi

Fi

# End

Run open-source software on MapR: excellent support

Some may worry about using a "hybrid" such as MapR. I mean most of the tools you use will become open-source software, of course, the file system will not. This is a challenge for some people in the open-source community, because some people will think, "I want to run Spark; I want to run programs such as Mesos; if I want to run programs such as MapR at the same time, who will provide me with support? Who will help me make it work? If I run it on standard Apache HDFS, most people will not be able to get help, from a community perspective ." This is one of the fears when people bind open source and closed source.

However, the example I found is that MapR works well with the community through resources such as answers.mapr.com and direct interaction. If there are some issues that I cannot solve, because the code I need does not exist, mapR is always happy to work with me and help me understand what will happen.

Some suggestions for people who want to run Mesos and Docker on MapR

At first, a large number of resources were determined to be allocated to MapR, and then the remaining resources were allocated to mesos. At present, I prefer it (half together) because I don't have an official installation package, I just installed MapR and mesos and said, "Yes, it works well together ." Things have worked well, because I cannot see any conflicts because it depends on how I call resources and some problems that MapR is working on, just as in the near future they will try to utilize resources dynamically between MapR and Mesos.

Other interesting projects that use Mesos on MapR

I can easily discuss topics for some subjects here! I am currently doing something very interesting about it-I can do anything on this cluster from running MySQL database to running my child's Minecraft server. All Minecarft world data is stored in MapR-FS through the NFS service. For me, it actually solves a problem because MapR can do something that other technologies cannot do. I don't know how to perform random read/write on HDFS files; I don't know how to make Minecraft run on HDFS-but I can implement it with MapR-FS.

As I mentioned, my home network is running on MapR-based Mesos because there are many interesting ways to use it. Of course, a few people will integrate at this level. I am doing this because I understand how MapR and Mesos work together. At the same time, I use an open-source linux-based DVR "MythTV", which enables you to record TV. I now run it in a VM, and my goal is to try to run it in the Docker in my cluster, just to see if I can implement it.

I really enjoy using technologies such as MapR-based Mesos and Docker, and I expect you to find that the code I wrote for persistent Docker storage is very useful.

Zhang Mingfeng, Senior devops and infrastructure engineer. Focuses on Distributed Systems and IDCs (systems, networks, storage, etc.) architectures. Have a deep understanding of systems, databases, networks, and storage. Daily development uses golang, c, c ++, python, shell. google fans.

Use Docker and seven commands to deploy a Mesos Cluster

Practical application scenarios of Mesos

Some Ideas about Apache Mesos

This article permanently updates the link address: