Apache mesos Communication architecture between modules

1. Introduce

The Mesos consists of four components, respectively Mesos-master,mesos-save,scheduler and executor, which are based on protocal buffer between the components actor Model for communication (using Open Source Library libprocess). That is, each module is a server (in fact, a socket server) that listens to messages from other modules, and when a message is received, the corresponding message handler function is invoked to handle it.

(What is Apache Mesos?) Reference: "Unified resource management and scheduling platform (System) Introduction", this article analysis based on Mesos SVN Revision 1327410.

2. Overview of communication frameworks

For a computing framework (such as Hadoop,spark, etc.), if you want to access Mesos, you need to write two components, Frameworkschduler and Frameworkexecutor, respectively. The two components implement the scheduler and executor interfaces, respectively, through the Schedulerdriver and Executordriver access Mesos, such as the black dotted line in the figure, which indicates that these components are related through function calls.

Other components, that is, mesos-master,mesos-save,schedulerprocess and executorprocess, communicate through the messaging mechanism (using the Libprocess Open Source Library).

(1) "Schedulerprocess and Mesos-master": Mesos-master allocates resources for each framework, which are passed directly to schedulerprocess and then called by schedulerprocess Frameworkscheduler related functions, which are assigned by Frameworkscheduler to tasks in the framework and returned to Mesos-master, forwarded by Mesos-master to the corresponding mesos-slave.

(2) "Schedulerprocess and Mesos-slave": If the Mesos-slave address is saved in schedulerprocess, the relevant message is sent directly to Mesos-slave. No more forwarding by Mesos-master.

(3) "Mesos-master and Mesos-slave": mesos-master management mesos-slave, such as monitoring slave health status, etc.

(4) "Mesos-slave and executorprocess": Mesos-slave is responsible for managing the executor of each framework and allocating resources for executor.

As shown in the following figure.

3. Message types and their processing functions

message types and processing functions of 3.1 Mesos-slave

message types and processing functions of 3.2 mesos-master

message types and processing functions of 3.3 Scheduler

Located in Sched/sched.cpp, implemented by Schedulerprocess/mesosschedulerdriver

message types and processing functions of 3.4 executor

In file Exec/exec.cpp, implemented by exectorprocess