New Understanding of Mesos design architecture

Author: Dong | Sina Weibo: XI Cheng understands | reprinted, but the original source and author information and copyright statement must be indicated in the form of a hyperlink. Website: dongxicheng. orgapache-mesosstudy-mesos-architecture-in-deepMesos contains four main services (actually a socketserver), which are

Author: Dong | Sina Weibo: XI Cheng understands | reprinted, but the original source and author information and copyright statement URL must be indicated in hyperlink form: http://dongxicheng.org/apache-mesos/study-mesos-architecture-in-deep/ Mesos contains four main services (actually a socket server), which are

Author:Dong| Sina Weibo: XI Cheng understands | reprinted, but the original source and author information and copyright statement of the article must be indicated in hyperlink form
Web: http://dongxicheng.org/apache-mesos/study-mesos-architecture-in-deep/

Mesos contains four main services (actually a socket server): Mesos Master, Mesos Slave, SchedulerProcess, and ExecutorProcess, which communicate with each other through Protocal Buffer messages, several Protocal Buffer message processors are registered in each service. Once a message is received, the corresponding message processor is called for processing. In addition to the above four services, Mesos also provides three programmable components: Alloctor, Framework schedtor, and Framework Executor. To write these components, you must implement several interfaces as required, these interfaces are called by adjacent services respectively.

Most people think that the Framework must be a common Framework, such as MapReduce, Storm, and Spark, after seeing the above Mesos architecture. The Mesos Master is responsible for allocating resources to various frameworks, scheduler of various frameworks further allocates resources to various internal applications. This concept is wrong, and it is a wrong interpretation of the Mesos architecture.

In fact, the Framework can not only be a general Framework, but also a simple computing task like a Hadoop Job or YARN Application. That is to say, the Framework must be a "Framework ", A long-running service (such as JobTracker) can also be a short-lived Job or Application. If you want the Framework to correspond to a Hadoop Job, you can design the Framework schedtor and Framework Executor as follows:

(1) Framework Scheduler Function

Framework Scheduler is responsible for breaking it into several tasks based on the input data volume of the job, applying for resources for these tasks, and monitoring the running status of these tasks, once a task fails, you can apply for resources again.

(2) Framework Executor Function

Prepare the runtime environment for a Map Task or Reduce Task on a node, including preparing various jar packages and binary files, setting necessary environment variables, and necessary resource isolation, start Jetty Shuffle to provide remote data copy services for Reduce tasks, and receive and execute commands from Framework Scheduler (such as startup and killing tasks.

The above introduction shows that Framework Scheduler is only responsible for running a Hadoop Job. If you are familiar with YARN, you will find that it is exactly what MapReduce ApplicationMaster in YARN does, the design architecture of Mesos is so similar to that of YARN that we can easily run it in Mesos as a Framework schedmaster by modifying any ApplicationMaster of YARN.

Recently, Mesos provides a mesos-submit tool that allows your Framework schedmit to run on any Mesos Slave to prevent the client from running too many Framework schedmit, mesos's entire architecture and workflow have become almost the same as YARN.

To make it easier for you to understand the architecture similarity between Mesos and YARN, the following table lists the components of Mesos and YARN:

Components in Mesos	Components in YARN	Function
Mesos Master	Resource Manager	Entire ClusterResource management and scheduling
Mesos Slave	Node Manager	Single NodeResource Management (resource isolation, reporting, etc.), task initiation, etc.
Framework Executor
Framework Scheduler	ApplicationMaster	Single applicationThe basic operations include registration, resource application/acquisition, and resource allocation (internal tasks.

Since Mesos and YARN are so similar, which one should we use? Or, which system is more promising?

YARN has obvious advantages in the following aspects: (1) Large manpower investment. Currently, YARN has a dedicated company (hortonwork) for maintenance and development (2. YARN evolved from Hadoop 1.0 and inherited Hadoop's popularity. A large number of companies and developers shared patches. However, Mesos has a simple design and easy to use. Unlike YARN, a resource allocation process involves several state machines, more than a dozen States and more events. However, in terms of stability, both systems are in the R & D and testing stages, and there is still a gap between stable and available systems.

Original article, reprinted Please note:Reposted from Dong's blog

Link:Http://dongxicheng.org/apache-mesos/study-mesos-architecture-in-deep/

Author: Dong, Author: http://dongxicheng.org/about/

Copyright©2013
This feed is for personal, non-inclucial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)