How we reduced the server from 30 to 2: Migrating from Ruby to go language (GO)

This is a creation in Article, where the information may have evolved or changed.

We developed the first version of the Ironworker 3 years ago, is written in Ruby, the API based on the development of rails. It didn't take long for us to grow to a fairly large scale, and soon we reached the top of the Ruby program's load limit. To make a long story short, we switched to the go language, read on, and here's how things go.

The original design

First of all, do a little background: we developed the first version of Ironworker, originally called Simpleworker (very good name, isn't it?) ), using Ruby. We used to be a consulting firm to develop applications for other companies, and at the time there were two of things that were fired very well: Amazon Web services and Ruby on Rails. So we've developed applications that are based on the AWS Ruby on Rails architecture and therefore attract a lot of big customers. Our intention to develop ironworker is to source our own needs. We have a lot of customers who do hardware equipment, they will send us the data for 7x24 hours, I need to collect this data, to organize them into useful information. A typical practice is to have timed tasks traverse the data every hour of the day. We thought that we should develop a thing that could handle all the users ' data without having to do a large number of scheduled tasks for each customer alone. So we developed a service class application and used it internally for a while, but then we thought there would be other people needing the application, so we decided to publish it, so that Ironworker was born.

The CPU usage that our servers can tolerate is probably 50-60%. When exceeding this amount, you need to increase the server to keep it around 50%. As long as we don't mind a lot of server rental fees (of course we do), this model will work very well. But the biggest problem is when there is a massive spike in traffic. When a large traffic spike arrives, it produces a domino effect that will drag down our entire cluster of servers. When certain indicators exceed the threshold value of 50%, our rails server consumes 100% of the CPU usage and becomes unresponsive. This causes the Load Balancer device to assume that it is down and move it out of the distribution pool so that the load on the unresponsive server is transferred to the other servers in the pool. Because the remaining servers in the pool need to host the load on this lost server and the traffic spikes, there will inevitably be a second server down, and the load balancer will remove it, wave upon wave. Soon all the servers in the pool will be exhausted. This phenomenon is also called Colossal Clusterf**k (ref: +blake Mizerany).

Here is a simple drawing that depicts the domino outage effect.

The only way to avoid this happening in this architecture is to keep a lot of extra processing power, so that the load on our servers is much lower than it should be able to afford, but that means spending a lot of money. It has to change that state.

overriding an app

I decided to rewrite the application. This is an easy decision, and it is clear that our Ruby on Rails cannot support the growth of our business. We have had many years of experience developing Java and have written a lot of things that require very little resources to handle loads, much more than Ruby on rails, and I know we can make a lot of improvements. So the next question becomes what language should be used?

Select a language

I am open to any new advice and I can go back to Java. Java is a great language in many ways (such as performance) (yes?). ), but after years of writing the Ruby program, I've been obsessed with its development efficiency. Ruby is fun, plain and simple.

We searched for a better scripting language than Ruby (Ruby is not very bad), like Python and Javascript/node, and we looked at the derived languages of Java, such as Scala and Clojure, and other languages such as Erlang ( AWS uses it) and the Go Language (Golang). Go language wins . In fact, its concurrency features as a base component are too strong; its standard core library provides everything we need to develop API services; it's simple, it's fast to compile, it's like Ruby, thego language is interesting ; Finally, numbers don't lie. After a prototype and a performance test, we learned that we can make a significant improvement in load capacity through it. After consulting the team ("This is good, Google support behind it"), we played battle.

It was a risky decision at first when you decided to gamble on the go language. The Go language Community does not have much to form, there are not many open-source go language projects, there is not much success in using the Go language on official products (is there?). )。 And we're not sure if we can get the top talent in this area after we identify the go language, but soon we find that we're able to recruit top talent-precisely because we chose the go language. We are the first company to publicly announce the use of go in our products, the first company to post the Go language job vacancies in the Go Language mailing list. Many top programmers want to come to us because they can use the go language in their daily programming.

The performance of the Go language

After we launched our first go language version, our number of servers was reduced from 30 to 2, and only 2 servers were reserved for redundancy. They're like they're not being used at all, just like there's no program running on them. Our CPU usage is less than 5%, and the entire application runs on only hundreds of KB of memory (at startup), compared to 50MB for rails applications. This comparison even includes the use of virtual machine memory! This is really the difference between heaven and land. Since then we have not experienced too much of the accident of Milford downtime.

Our business has grown a lot since the

. We have more traffic, we have added two new services (ironmq  and  ironcache), and we have hundreds of servers to support our customers ' needs. This is all done with go for the background motor. In retrospect, choosing the go language is a smart move that allows us to develop better products, help companies grow, expand their size, and attract top-notch talent. I believe that it will continue to help us to improve in the foreseeable future.