Uncover the distributed cloud computing framework you don't know Mesos

Mesos Computing Framework a cluster manager, which provides efficient, resource isolation and sharing across distributed applications or frameworks, and can run Hadoop, MPI, hypertable, and Spark. Use zookeeper to implement fault tolerant replication, isolate tasks using Linux containers, and support multiple resource scheduling allocations.

The Mesos contains four main types of services (actually a socket server), which are Mesos Master,mesos slave,schedulerprocess and executorprocess, respectively. They communicate through protocal buffer message, each service registers several kinds of protocal buffer message processor, once received some kind of message, then call the corresponding message processor to handle. In addition to the above four services, Mesos also provides three programmable components, respectively, Alloctor, Framework Scheduler, and framework Executor, which require several interfaces to be written, These interfaces are called by the next service in the diagram below.

When most people see the above Mesos architecture, they assume that the framework must be a common framework, such as MapReduce, Storm, Spark, and Mesos Master is responsible for allocating resources to the various frameworks, The scheduler of each framework further assigns resources to the individual applications within them. This idea is wrong and a wrong interpretation of the Mesos architecture.

In fact, the framework can be not only a generic framework, but a simple computational task like a Hadoop job or a yarn creator, that is, the framework needs to be a "framework", or a long-running service (such as Jobtracker, for example), or a short life cycle job or creator. If you want the framework to correspond to a Hadoop Job, you can design the framework scheduler and framework Executor:

(1) Framework Scheduler function

The Framework Scheduler is responsible for breaking into several tasks according to the amount of input data in the job, requesting resources for these tasks, monitoring the running status of these tasks, and then requesting resources again if a task fails to run.

(2) Framework executor function

Prepare a running environment for the map task or the reduce task on one node, including preparing various jar packages, binaries, setting the necessary environment variables, making the necessary resource isolation, initiating jetty shuffle to provide remote data copy services for the reduce task, Receive commands from the Framework scheduler (start task, kill task, etc.) and execute.

As you can see from the above introduction, the Framework Scheduler is only responsible for running a Hadoop Job, and if you are familiar with yarn, you will find that it is yarn mapreduce applicationmaster do, yes, Mesos and Yarn's design architecture is so close that we can easily modify any of yarn's applicationmaster to run it as a framework scheduler in Mesos.

The recent Mesos provides a mesos-submit tool (note that the tool is not yet complete) that allows the user's framework scheduler to run on any mesos slave to prevent clients from running too many frameworks Scheduler, in this way, Mesos's entire architecture and workflow have become almost yarn.

To make it easier to understand the architectural similarities between Mesos and yarn, the following table shows the Mesos and yarn component tables:

Component functionality in the component yarn in

mesos mesos Masterresource Manager Resource management and scheduling for the entire cluster Mesos Slavenode Manager Resource management for individual nodes (resource isolation, reporting, etc.) Task Launch framework executorframework schedulerapplicationmaster management and resource scheduling for a single application, including registration, resource request/acquisition, resource allocation (for internal tasks), and so on.

Since Mesos and yarn are so close, which one should we use? Or which system is more promising?

At present, yarn in the following several aspects of the obvious advantages: (1) a large human investment. At present yarn has specialized company (hortonwork) Maintenance and development (2) high visibility. Yarn, which evolved from Hadoop 1.0, inherits the popularity of Hadoop and has a large number of companies and developers sharing patch. However, the design of Mesos's greatest advantages is simple, easy to use, it is not like yarn, a resource allocation process involves several state machines, and each state machine more than 10 states, more than 10 kinds of events. However, the stability of the two systems are in the research and development and testing phase, from the stable available for some distance.

you might also like: 1 Apache mesos overall architecture 2 Apache Mesos underlying base 3 Apache Mesos Inter-module Communication Architecture 4 Apache Mesos dispatch mechanism 5 Apache Mesos task status update process analysis 6 Apache Mesos Task allocation Process Analysis 7 Mesos framework and executor registration process