There are more than one server, and there is a reason to try the cloud operating system.

There are more than one server, and there is a reason to try the cloud operating system.

Mesos is often well known for its mature cases in large-scale production environments. But in fact, Mesos is also very useful for internet startups. The most notable point is that the Mesos-based cloud operating system was used from the very beginning to enable an IT architecture to support the rapid development of businesses at different stages. And let the technical staff focus more on the business, rather than focusing on server configuration. With Rahman's statement, "as long as you have more than one server, there is a reason to try the cloud operating system ."

 

American startup Mattermark uses Mesos Technology

Mattermark is a highly popular startup dedicated to data delivery for private enterprises. As a startup company, its IT infrastructure is relatively small. However, thanks to this initial feature, it has also been able to smoothly get rid of the operational inefficiency problems that are hard to solve by many larger and more mature enterprises. Like many large enterprises, Mattermark uses the Apache Mesos and Marathon frameworks to solve their own unique problems.

This young company, just two years old, only runs some Amazon Web Services instances that store GB-level data, but thinks that their IT architecture has become indispensable, it will affect business development in the near future. In this regard, the most prominent practical problem is data processing tasks, especially Mattermark's unavoidable massive data mining, machine learning, and indexing in daily operations-past, these tasks have been carried out in a casual manner without adequate theoretical guidance.

"We have a lot of EC2 instances running important work content, but no one knows about the specific operating mechanism," said Samiur Rahman, a machine learning engineer at Mattermark. "This is really troublesome ."

The management of the company realized that to really provide reliable data sources for private enterprise customers, it was necessary to rectify the existing operating mechanism. "We know that the company will continue to expand in the next one to two years," Rahman said. "So we either continue to work in the current environment and build it, or, as other companies do, completely remove existing facilities and move them directly to modern infrastructure ."

 

Use Mesos for infrastructure Scheduling

Mattermark decided to use Mesos to redesign its infrastructure a few months ago, and provided several special requirements that must be met for the new system:

• An abstraction layer needs to be established between developers and the company's AWS instances.
• It can distribute tasks to different AWS instances.
• Resources can be allocated to a specific task based on its actual needs.
• Task-oriented scheduling implements a highly refined control mechanism.
• Isolate resources to avoid competition for resources from adjacent instances.

The company currently runs Chronos and Marathon on Mesos, and this has brought about an ideal return. Mattermark can now schedule tasks in batches in a controlled manner and intelligently run different types of tasks in the same resource pool to improve AWS resource utilization. For example, it can run multiple low-resource Web page acquisition tasks and other machine learning tasks with high memory and CPU usage in the same instance.

"It is very important to be able to effectively allocate such resources (that is, to allow these workloads to coexist on the same resource)," Rahman pointed out. In this way, Mattermark can not only automate task processes in a reliable manner, but also reduce the usage of AWS instances without changing the work effect and thus reduce the use cost.

Compared with the best server monitoring practices in the past, enterprises need to consider the countermeasures when the resource usage approaches 100% in the initial stage. "Our monitoring mechanisms are different, because 80% to 90% is the ideal resource utilization rate, "he explained. "This means we can make full use of existing resources ."

Although Mattermark still has some workloads running on AWS bare metal instances supported by Elastic Load Balancer (that is, non-Mesos worker nodes ), but Rahman stressed that tasks that require high availability are already running in the Mesos-Marathon environment. As Mattermark constantly introduces new workload and task types, it will all be carried by the new system.

"We are committed to running all kinds of workloads and tasks in Mesos," he pointed out. Mattermark has begun to widely use pre-configured Docker containers as the application running environment, currently, it can migrate various processes from pure AWS instances to Mesos clusters running on AWS instances in a relatively simple way.

Meet the Challenges of big data with a positive attitude

But in the case of Mattermark, the real value of Mesos is to allow it to keep its infrastructure scale and business growth expanding at the same time-and vice versa. In other words, let IT resources grow smoothly with the development of the business, so that IT architecture will not become the bottleneck of business development, nor will IT be wasted because of the need to reserve a large number of IT resources.

Although it is difficult to define a MySQL database of only 150 GB as "Big Data", Rahman said Mattermark plans to significantly expand its size over the next few years. One of the most prominent reasons is that the longer the Mattermark company takes, the more data it needs to store in the database for each enterprise customer.

But from a more strategic perspective, Mattermark hopes to expand its database coverage from 1 million companies to hundreds of millions of enterprises around the world. As the number of enterprise customers increases, it certainly wants to provide each customer with a richer data type, including the number of employees, website traffic, financing information, related news, and social media attention.

"The scale of our business depends on the speed at which we update data and the ability to include more enterprise customers into the database," Rahman pointed out.

In his view, Mesos can help from multiple perspectives, making it easier for Mattermark to introduce various necessary new data processing technologies, including Kafka and Spark, and quickly add the corresponding capacity when running a large-scale processing task. All in all, the less effort the company has invested in the infrastructure and task fit, the more powerful it is to focus on accessing, analyzing, and delivering the right data at the right time.

"We hope that the operating system will always be ready for scale scaling to help us get more data and extract more valuable information from it," Rahman said. "In addition, we also hope to continue to deliver data to our customers ."

 

Start-ups also need to be brave enough to try Mesos or data center operating systems

Looking at the future, Rahman said he wanted to see Mattermark migrate its operating system from open-source Mesos to the Mesosphere data center operating system (Chinese friends can choose cloud users ), to a large extent, this means to get rid of the troubles caused by software component updates and vulnerability fixes. Although Early experiments are still being conducted on DCOS Early Access, he can set up a system that usually takes weeks to build within 30 minutes.

"Start-ups should understand DCOS because it makes operations easier," he explained. In addition, startups do not have to worry about trying DCOS or open-source Mesos components because they are not as large as Yelp, apple, or Twitter.

"Mesos's natural developer free space and sound operational efficiency have rewarded us with a tangible return from system redesign," Rahman concluded. "As long as you have more than one server, there is a reason to try it out ."

 

DCOS in China "dtplus cloud"

Dtplus cloud is a cloud Operating System (data center operating system) developed by MSOs and Docker technologies. It enables users to manage cloud applications and clusters like a single computer. In this way, enterprises can focus more on their own business, rather than focusing on IT infrastructure construction.

Finally, I used Rahman's statement: "As long as you have more than one server, there is a reason to try the cloud operating system. "

This article permanently updates the link address: