Mesos Framework and Executor registration process

Taking the Hadoop framework as an example, this paper introduces the process of registering the framework and executor to Mesos.

1. Framework registration Process

(1) When Jobtracker starts, it invokes the Mesosscheduler start () method

(2) The Mesosscheduler start () method creates a Mesosschedulerdriver object and passes itself as an argument to the object.

(3) Mesosschedulerdriver initialization, creating a Schedulerprocess object

(4) Mesosschedulerdriver initialization, calling Masterdetector::create (), which sends a newmasterdetectedmessage message to the Schedulerprocess object

(5) When the Schedulerprocess object receives a newmasterdetectedmessage message, it sends a registerframeworkmessage message to master

(6) Master receives the message, saves the relevant information, and returns the Frameworkregistedmessage message confirming the framework registration success

2. Executor Registration process

This section describes the process by which the framework Frameworkx registers a executor Executorx on a slavex:

(1) master sends a message to Slavex for the first time to execute a task in Frameworkx runtaskmessage

(2) Slave receive the message, run the corresponding message handler function Runtask ()

(3) the Isolationmodule Lauchexecutor () function is invoked when the function finds that the frameworkx corresponding Executorx is not started on the slave

(4) The function creates a Frameworkexecutor object and invokes the Executorprocess Initialize () function to initialize and start the Tasktracker

(5) The Initialize () function creates the message registerexecutormessage and sends it to slave

(6) Slave receives the message, invokes the object's message handler function Registerexecutor, which creates the Executorregisteredmessage message and returns it to the Executorprocess

(7) Executorprocess receives the message, calls the corresponding message handler function registered (), which further calls the Frameworkexecutor registered () function

Next, the flow of the runtaskmessage message that master sent to slave is shown in the following diagram. It should be noted that for the same computing framework, Mesos creates only one resource container on a slave, all tasks run in this container, that is, Mesos cannot do task-level isolation, Only executor level isolation is possible, and for the same frame, all tasks on the same slave run in one executor.

For Hadoop, only one tasktracker is created on each mesos-slave, and the Tasktracker is placed in a executor (corresponding to a Linux container). All tasks on the same tasktracker are in the process tree where the tasktracker is located, thus sharing the corresponding resource for the executor. When Tasktracker receives a new task, it increases the amount of resources that the executor can use (using "Lxc-cgroup–n%s%lld") and reduces the amount of resources available to the executor when a task runs complete (using the Lxc-cgroup–n%s%lld ").