Apache Mesos underlying base library

1. Kyoto Buffer

Protocal buffer is Google open source for data exchange library, often used for cross-language data access, the role played in general for the object serialization/deserialization. Another source of similar Open-source software is Facebook's Open-source thrift, where the two biggest difference is that thrift provides the ability to automatically generate RPC and protocal buffer needs to be implemented by itself, but one advantage of protocal buffer is its serialization/ Deserialization is very efficient.

2. libprocess

Libprocess is an efficient messaging programming model (based on message delivery, rather than RPC), written by C/s + +, open source by Berkeley. The whole implementation is very simple, including the most basic message sending and receiving.

2.1 Libprocess Model

In the Mesos, there are mainly four roles, namely: Mesos-master,mesos-slave,framework (HADOOP/SPARK/MPI, etc.) scheduler, Executor (the component that executes the framework task on mesos-slave), each role is a process that inherits the Protobufprocess class in Libprocess (it inherits the Process Class), In this way, they are programmed into a socket server that runs in the background and constantly listens for protocal buffer messages, as shown in the following illustration:

2.2 Various common functions

The Libprocess+protocol buffer combination is the Mesos most important messaging base (no RPC mechanism), and is highly efficient because the library uses a communication mechanism based on protocal buffer messaging. Mesos two common header files are libprocess\include\process PROCESS.HPP and PROTOBUF.HPP, these two provide APIs for messaging, where PROCESS.HPP is the core file that provides the original interface, And PROTOBUF.HPP is based on PROCESS.HPP, adding Protocalbuffer object parameters, make Protocalbuffer easy to use.

(1) Install

void Install (void (T::* method) (p1c), P1 (M::* param1) () const);

Installs a handler that handles Protocalbuffer messages, where the message type is M, and the corresponding handler for the message is method, and the function parameter is m::* param1. Example: Mesos slave/slave.cpp:

Install (&slave::newmasterdetected, &newmasterdetectedmessage::p ID);

Install a Handler,mesos slave processing newmasterdetectedmessage (Protocalbuffer object) Once the message is received, newmasterdetected function processing is invoked. The input parameter of the function is the PID attribute in the Newmasterdetectedmessage message.

void Install (const std::string& name,void (T::* method) (const upid&, const std::string&))

Installs a handler that handles strings, that is, when the string name is received, the function method is invoked to process it. This API maintains the heartbeat between master and slave in typical applications in Mesos to make sure that each other is alive:

In Slave/slave.cpp:

Install ("PING", &slave::p ing);  void Slave::p ing (const upid& from, const string& body) {Send (from, "PONG");}

In Master/master.cpp:

Install ("PONG", &slaveobserver::p Ong);     void Pong (const upid& from, const string& body) {timeouts = 0;   pinged = false;         } void Timeout () {if (pinged) {//So we haven ' t got back a pong verb ... if (++timeouts >= max_slave_timeouts) {         Deactivate ();       Return;     } send (Slave, "PING");     pinged = true;   Delay (slave_pong_timeout, self (), &slaveobserver::timeout); } 

(2) Send

void Send (const process::upid& to, const Google::p rotobuf::message& message)

Sends a message to a upid where upid represents a socket containing both IP and port information, and message messages are Protocalbuffer defined objects.

(3) Dispatch

void Dispatch (const upid& PID, const std::tr1::shared_ptr >& f)

Execute function f in process PID, in order to improve efficiency, the function does not wait for function f to complete, but instead uses asynchronous method: Put function f into a function queue, by another process (or more) continuously from the queue to get the function, sequentially executes.

(4) Delay

Timer delay (double secs,const pid& pid,void (T::* method) ())

Delay secs The method in the PID of the scheduling process and returns a counter through which the schedule can be canceled.

In Mesos, an infinite loop is cleverly constructed through this function to continuously detect idle resources and assign them to each framework, as follows:

void Master::initialize () {... timerticktimer = delay (1.0, self (), &master::timertick);} void Master::timertick () {... Timerticktimer = Delay (1.0, self (), &master::timertick); } 

The function code snippet above can complete the function of calling the Timertick function every 1s.

3 Boost

Very well-known open source C + + base, the inside of the STL is very efficient and convenient, has been used by many well-known software.

4. Zookeeper

It is a reliable coordination system for large distributed system, which includes: Configuration maintenance, Name service, distributed synchronization, group service, etc. Mesos uses zookeeper to solve master single point of failure, use zookeeper to build a master cluster, and when master fails, choose a standby master to master.

5. Glog

Google Open source C + + log library, the main use of C + + program print log, printing format is as follows:

I0411 17:26:54.150193 20653 main.cpp:111] Creating "process" isolation module

I0411 17:26:54.150400 20653 main.cpp:119] build:2012-04-11 16:50:21 by Root

I0411 17:26:54.150658 20653 main.cpp:120] starting Mesos slave

I0411 17:26:54.152981 20669 slave.cpp:191] slave started on 123.145.2.2:34694

I0411 17:26:54.153024 20669 slave.cpp:192] slave resources:cpus=2; mem=490

6. Gmock

Open source C + + unit Test framework

you might also like: 1 Apache mesos Overall Architecture 2 uncover the Distributed cloud computing framework you don't know