Practical application scenarios of Mesos

Practical application scenarios of Mesos

[Editor's note] the content in this article comes from a question on stackoverflow. The questioner wants to know which use cases Mesos has in actual use, twitter engineers answered questions from the perspectives of container orchestration, resource utilization, priority and resource preemption, and service operation.

I tried to find out why users are using Mesos. The following are some of the key points currently listed. I wonder if there are any other examples?

1) Let the user run all services in the same cluster rather than in a dedicated cluster (for example, through the terminal application cluster + DevOps mode, such as Jekins)

2) running different types of Mature Services (dev, test, production) in the same cluster does not know if this is feasible? You can use the lables label in Kubernetes to provide similar functions.

3) Mesos simplifies the use of traditional distributed services (such as Hadoop) by providing a series of operations such as simplified deployment, unified standard APIs, and resource packing.

Answer (Dave Lester ):

I am currently working on Twitter. I am also involved in the Apache Mesos and Autora projects. The following is my understanding of Mesos use cases.

The actual use of Mesos may be affected by the following factors: cluster size (from dozens of servers to thousands), available hardware (dedicated/static, or cloud/Scalable), and workload (providing basic services or batch processing services, or providing both services at the same time)

The points listed above make sense. Here I will add several new use cases.

Container orchestration

As the container environment (such as Docker) becomes more and more popular, many potential users are looking for a Mesos + scheduler solution to manage image orchestration after creation. Mesos has been quite mature so far and has been proven to provide services in a large scale. I think Mesos solutions are much better than some temporary solutions.

Improve resource utilization

For companies that use more than 50 servers, a common motivation for using Mesos is to increase resource utilization and reduce O & M costs. There are already many such companies, such as various public cloud and private cloud service providers. In Ebay's case, they used to run Jekins on Mesos to reduce the use of virtual machines. Mesosphere also published related articles on HubSpot (running on AWS) case studies, this article describes how HubSpot uses dozens of large servers to replace hundreds of small servers, making the hardware more efficient.

Priority and resource Preemption

On Twitter, we run the Mesos service through an Apache Aurora scheduler. In this scenario, we can improve resource utilization by running different mature services on the same cluster. Aurora has an overview of environment variables, which allows users to run services of different environment types, such as production, development, or test. Aurora also has a built-in priority preemption mechanism that allows it to prioritize tasks in the production environment. When these resources need to be used by tasks in the production environment, the system can terminate non-production tasks in a timely manner. In each environment, a similar priority system exists.

For a long time, the function-related resource preemption function will also exist in the Mesos kernel. This is a killer feature, which can enhance resource utilization, and run different mature services in different environments (dev, test, prod. If you are interested in related topics, you can refer to MESOS-155 and MESOS-1474.

Run both the batch processing service and common service

You can run batch processing tasks and other common services in a shared Mesos cluster, which plays a key role in improving resource utilization. Let's review the Myriad project. This project wants to put the jobs of Mesos and YARN in the same cluster. At this time, the user does not care about the large-scale deployment of batch services and common services at the same time. It will be easier to run multiple frameworks in a shared cluster, this is also a direction of community development.

Use Docker and seven commands to deploy a Mesos Cluster

This article permanently updates the link address: