Why we choose Golang to reconstruct the worker system

This is a creation in Article, where the information may have evolved or changed.

The cause of the story is here.

Before posting a post, talking about the Golang reconstruction of the worker system, a lot of friends asked the question of language choice.

In fact, we did a lot of research before rewriting our worker system with Golang.

What really made us determined was a parse article: how we Moved our APIs from Ruby to Go and Saved our sanity. The article says why Facebook's parse team chose Golang instead of Ruby.

I translate the key points:

The problems that parse faces

The parse is similar to the burst technology stack: Server Unicorn, deployment using Capistrano. Many of the problems in front of high traffic are exponentially magnified, and the app server restarts for a long time each time it is deployed. And Unicorn's reboot is not really ' graceful ' (which we also share, the service will be interrupted for about 10 seconds after restarting, and parse has written a library ' Grace ' to solve the problem of restarting the interrupt service after using the Golang refactoring). The impact of this disruption of service is very bad.

In addition, like unicorn, each process can handle only one request at a time, not just the extreme waste, but also the entire worker poll if an action suddenly slows down. Causes the entire service to be unavailable.

Solution Selection

Parse did a comparison before eventmachine,jruby,c++, C #, Golang, and finally chose go.

Eventmachine

Parse uses Eventmachine to implement their push service, which, in the course of use, always encounters some strange bugs due to the lack of maturity of the associated gem. is also an ecological problem, leading to the realization of certain functions when no wheels are available.

Jruby

Parse is now a ruby implementation, so JRuby is the right choice? JRuby based on RVM can handle a large number of requests concurrently and looks very good. No! JRuby lacks support for various asynchronous libraries. Parse fears a second refactoring to cope with business growth: from JRuby to Java. And the parse team of engineers did not want to deploy and tune various parameters in the JVM.

It also mentions the Twitter team's survey of JRuby before moving to Scala: Twitter did a lot of work on Mruby, including writing a GC tool. Originally in Mruby work very good, high efficiency of the library, to the JRuby on it is not good (plainly is a variety of library immature, the ecosystem is too important!) Although the JRuby itself was twice times faster after the migration, other things were facing a variety of problems, a little outweighed by

That's not a fault of JRuby; It's just in the moment the surrounding ecosystem is still kind of immature. Cruby ' s was too; We put a lot of investment into it, which we can now take advantage of, and we would has to does the same for JRuby.

Storm-Diffuse team to tell the truth no one has used JRuby, and after such a long period of development, the ecology should be much better, but despite this, there is still a lot of work to do after the migration?

C++

The parse team has a lot of C + + development experience, but C + + code is difficult to debug and maintain. Personally, I seriously feel that C + + is definitely not a Web project choice. There is also a lack of web-related library support.

C

C # has very good concurrency model support. But on Linux ... Let's take a look at the next bar.

Golang

Golang language efficiency, language level support concurrency, syntax is very simple and easy to get started, concurrency model easy to understand. (We reconstructed only one hours of grammar for the team and then gave some good workers as a reference, and then everyone could reconstruct 2-3 workers smoothly, in two weeks time). should be the best choice for the worker system.

Finally back to the problem of the comic strip.

Everyone's question more is that since all is IO consumption, why Golang will be so much faster. I try to explain (the level is limited): Golang static language does not need type inference discarded a variety of syntactic sugars, in the language efficiency level of a lot faster, in addition to the database IO Gorm no ActiveRecord black magic, nature will be much faster.

The burst worker system bottleneck is at high concurrency peaks, which accumulate after the resistance is not over. While Golang is only 5 times times faster on a single task, a good concurrency mechanism makes job execution fast. And in the original system each machine 150 concurrent running slow, some 100ms tasks are waiting for 23s long.

A single task executes 5 times times faster and 5 times times faster, so it is reasonable to reduce from 10 to 1 units and Golang machines.

In refactoring, parse considers scenarios that can accommodate up to 10 times times the current peak of the business. I think we have to be aware of this when we choose our options. While some solutions can really solve the current dilemma, it is beneficial to make adjustments to future architectures, or to say compatible