Apache Mesos Overall Architecture

1. Foreword

Like most other distributed systems, Apache Mesos, in order to simplify the design, also uses the Master/slave structure, in order to solve the master single point of failure, master to do as lightweight as possible, the above all the number of dollars It can be reconstructed through various slave, so it is easy to solve the single point of failure by zookeeper.

(What is Apache Mesos?) Reference: "Unified resource management and scheduling platform (System) Introduction", this article analysis based on Mesos SVN Revision 1327410)

2. Basic terminology Interpretation in Apache Mesos

(1) Mesos-master:mesos master, mainly responsible for the management of various frameworks and slave, and the allocation of resources on the slave to the framework

(2) Mesos-slave:mesos Slave, responsible for managing the various mesos-task on this node, such as: allocating resources for each executor

(3) Framework: Computational framework, such as: Hadoop,spark, via Mesosschedulerdiver access Mesos

(4) Executor: Actuator, installed on Mesos-slave, for starting tasks in the calculation framework.

When a user attempts to add a new computing framework to Mesos, a framework scheduler and executor are required to access Mesos.

3. Overall structure

The Apache Mesos consists of four components, mesos-master,mesos-slave,framework and executor respectively.

Mesos-master is the core of the entire system, responsible for managing the various frameworks for access Mesos (managed by Frameworks_manager) and slave (managed by Slaves_manager), The resources on the slave are assigned to the framework (managed by the standalone plug-in module allocator) according to some policy.

Mesos-slave is responsible for receiving and executing commands from Mesos-master, mesos-task on management nodes, and allocating resources for each task. Mesos-slave sends its own resources to Mesos-master, and the allocator module in Mesos-master decides which framework to allocate resources to, currently considering two kinds of CPU and memory, that is to say, Mesos-slave will send the number of CPUs and the amount of memory to Mesos-master, and when the user submits the job, you need to specify the number of CPUs and the amount of memory needed for each task, so that when the task runs, Mesos-slave puts the task on the Linux that contains the fixed resource. Container to achieve the effect of resource isolation. It is obvious that Master has a single point of failure, and for this reason, Mesos uses zookeeper to solve the problem.

The framework refers to the external computing framework, such as Hadoop,mesos, which can be registered to access the Mesos to Mesos for unified management and resource allocation. Mesos requires that an accessible framework have a scheduler module that is responsible for task scheduling within the framework. When a framework wants to access mesos, it needs to modify its scheduler to register with Mesos and get the resources that Mesos assigns to them, so that the resources are assigned to tasks in the framework by their own scheduler, that is to say, The whole Mesos system adopts a Double-layer scheduling framework: The first layer, the resource is allocated to the framework by Mesos, and the second layer, the framework's own scheduler allocates resources to its own internal tasks. The current Mesos supports three languages of the scheduler, respectively, C++,java and Python, in order to provide a unified access to various dispatchers, mesos internal use of C + + to implement a mesosschedulerdriver (dispatcher drive), The scheduler of the framework can invoke the interface in the driver to interact with the Mesos-master to complete a series of functions (such as registration, resource allocation, etc.).

Executor is primarily used to start tasks within the framework. Because of different frameworks, the interface or the way in which the task is initiated, when a new framework is to be connected to Mesos, a executor should be written to tell Mesos how to start the task in the framework. In order to provide a unified way of executing the actuators to various frameworks, Mesos internally implemented a mesosexecutordiver (actuator drive) using C + +, and the framework can tell Mesos how to start a task through the relevant interface of the drive.

You might also like: