Lessons from slideshare-how to avoid cloud computing

This article is translated from a gigaom article. The original post address is Jonathan boutelle, one of the CTO and founders of the famous site slideshare.

Cloud computing is a powerful tool for startups, because they can have almost unlimited computing capabilities with just one click, and these computing capabilities can well create new opportunities. It is very powerful to start or shut down thousands of servers with a single mouse click, but as the comic book taught us, "great power comes great responsibility (more powerful, more responsible) ".

Our company slideshare has used cloud computing in almost all of our affairs, which also leads to some big mistakes in using cloud computing. Below are two of the most obvious examples:

$ Five thousand is wasted before trial.

A few months ago, we became very fascinated by hadoop. We even organized a hadoop hacking day in the office ), and quickly write some hadoop prototype code to analyze slideshare user data,

Hadoop analysis is an extremely suitable task for cloud computing. Although you need a lot of computers, it takes only one day to process all the data. But when we started to use more and more real datasets to test our prototype code, it began to spend more and more time to complete a task.

At that time, I decided to quadruple the number of machines (from 20 to 75 ). This decision is very meaningful. If a task can be completed only when it takes 100 computations, then 100 machines can be completed in just one hour.

Several hours after I made this decision, a large site accident caught the attention of all engineering team members. To solve this accident and other related accidents, we work for one night and one whole day until Friday afternoon. After a weekend of peace of mind, we found that the hadoop analysis task was still running before the accident on Monday. The code that contains bugs fails in a way that we did not anticipate, so that even if we add more hardware to this problem, this problem cannot be solved. At the same time, we received a bill of $ five thousand from Amazon Web service.

Our lesson is: if you really want to use the power of cloud computing, You need to constantly observe spending and make sure it doesn't get messy or out of budget, especially when you scale out and scale down cloud computing. Unfortunately, Amazon Web service does not provide any reminders or chart tools to help users easily track expenditures. Although tracking expenditures are a CSV file involving downloading, the complicated process of importing them into Excel for analysis, but it is indispensable.

Difficulties in using cloud storage

We recently found that our spending on storage (S3) has increased dramatically. After several days of investigation, we found that there are no clear principles for using storage. For example, some files that can be deleted are retained; different types of files are placed in the same directory; and some files do not know their source and whether they are needed.

Amazon S3 and other similar cloud storage can be considered as a large file system without any control over the data location, it is used by users to ensure that the storage is used in a systematic manner. If a person writes code, it is very simple, but when a team writes multiple programs dependent on cloud storage, it is easy to forget to delete some files. You need to ensure that you do not waste storage. The only way is to clearly define where the data is stored. One best practice is to place different types of resources in different "buckets (directories at the highest level of the bucket and S3)", this is the only way to get space for each type of data.

Principles of spider

In the above two examples, we know that we are not using the power of cloud computing very strictly. If we have used hardware before, we will also touch on hardware restrictions (for example, disk space used up), this is a troublesome thing, but it forces us to summarize the past behavior to spend more reasonably. It is a good thing to have a powerful cloud computing power, but if you want to use it, you must have a certain sense of responsibility.