AWS helped dummi studio deploy HPC projects at a low cost for Scientific Research

About dummi Studio

Dummi studio is a research group organized by students from the Chinese Emy of sciences and beiyou Research Institute to complete cryptographic research projects jointly undertaken by the Chinese Emy of sciences and beiyou Research Institute. The group focuses on trying and practicing the latest and coolest technology. At present, this group has made various explorations in mobile Internet, cloud computing, social networks, LBS and other fields.

Challenges

As a team of graduate students, dummi sdudio provides support for the development of computer science through research and verification of cutting-edge computer technologies. Dummi studio has many scientific research projects and many are concentrated on gridded HPC applications. As the project progresses, the research team needs to be able to deploy the HPC lab project at any time based on the study progress. From the perspective of cost, it is difficult for the team to purchase servers for a single project or rent fixed equipment for computing experiments due to limited funding for individual scientific research projects. From the perspective of human resources, this is a group organized by students who focus on the promotion of various scientific research projects without dedicated Logistics support teams, such as IT support personnel, provide IT infrastructure support for the implementation of each project; it implementation requires each research project team to promote and manage it by themselves. Therefore, for various research teams of dummi studio, they need to find a service provider to meet their high-performance computing cluster needs, it also needs to be able to control costs well and complete project experiments in a relatively short period of time.

Why use AWS

As a research team, dummi studio mainly considers computing experiments for implementation projects from the following aspects: cost, performance, ease of use, and response speed. After in-depth investigation and analysis, dummi studio believes that AWS can meet the needs of high-performance computing clusters required by project labs. At the same time, AWS provides spot price instance models, with low-cost features, you can also conveniently deploy experiments without having to purchase expensive machines for computation experiments. After comprehensive research and quick comparison of multiple cloud service platforms, dummi studio finally chooses to use the AWS cloud platform service. The related services mainly include: amazon EC2 On-Demand instance & spot instance, Amazon virtual private cloud, and Amazon simple notification service. When choosing AWS, dummi studio is mainly based on the following considerations:

Convenient cloud computing services and unique spot price instances
One of the basic services AWS provides to users is computing and networking services. Amazon EC2 is a Web service provided by AWS that provides an adjustable computing capacity in the cloud. Its web service interface is very simple, and users can easily obtain and configure the capacity. By using Amazon EC2, users can shorten the time required to acquire and start new server instances. When the computing requirements change, the computing capacity can be quickly expanded or reduced. In terms of computing service charges, Amazon EC2 charges fees based on the actual capacity used to ensure the effective cost. Amazon EC2 provides three billing modes: Reserved instances, on-demand instances, and spot instances. Spot Price instances are based on supply and demand and can be automatically determined. The user bids for the EC2 capacity that other users do not use in the system. The system generates a spot price based on the supply and demand relationship, and the system automatically determines which bidder is successful. For dummi studio, when their bid price is higher than the spot price, the system starts the instance; when the spot price is higher than the spot price, the system stops the instance; without considering the time, it offers the best price for dummi studio and ensures availability and continuity of instances, making it ideal for handling large-scale and dynamic workloads. The daily high-performance computing of dummi studio is characterized by fine-grained grid, which makes the implementation of computing applications less affected by the system stop instance.

No Operating System Limit
Because AWS has no restrictions on the operating platform, dummi studio can implement high-performance computing using GCC and avx assembly on SuSE/Amazon Linux, simplifying the work content.

Easy to implement
Dummi studio first attempted to deploy a program on an instance and achieved good results. Subsequently, dummi studio compiled scripts to try to batch deploy programs on 10 instances and test the performance. Then, the Group is divided into three directions: Performance Tuning for instances, performance optimization for programs, improvement and debugging of management script groups; during running, dummi studio continues to iterate the automated script group and adds new features such as detection usage, SNS alarm countermeasure, and automatic bidding instance application. Due to the ease-of-use of AWS and the efforts of our team members, it took only three days to officially run the entire project from the attempt to the AWS solution.

Benefits

Based on the deployment cost of 20 thousand yuan per dual-channel E5 Server, to complete project computing within three months, dummi studio needs to deploy about 500 machines at a cost of up to 10 million. Dummi studio only spent less than 0.5 million for AWS spot price instances. The cost of autonomous operation is about 20 times that of AWS services.

Yang Yaoyao, co-founder of dummi studio, said: "by using AWS spot price instances, we not only completed project computing within a limited time, but also saved a lot of research funds. The cost for using AWS services is as much as 20 times that for self-operation. We are very pleased with the correct choice we made ."

More information
For more information about AWS's solutions to educational institutions, see the AWS page: http://aws.amazon.com/cn/education/