Why am I moving from python to go

This is a creation in Article, where the information may have evolved or changed.

Should puppet Danale Yulaw's invitation, I went to the Xishan Habitat Ops team to do a short share and talk about why I was moving our project from Python to go.

Frankly speaking, in front of a bunch of Python users why it's a stressful thing to give up Python and go in with going, the language struggle is a perpetual, non-solvable topic, just like the Battle of Vim and Emacs, and a little less attention can cause fans to fight back strongly. So I will only start from the actual situation of our project, to tell why I finally chose go.

Why give up Python

First of all, I actually have to say why we chose Python. Before I joined the Enterprise Express team, the entire project, including the earlier Kingsoft Express, was developed using Python. As for why so choose, then the architect of the Onion told me, mainly because Python is easy to get started and develop quickly. Python is really a good choice for most of the people in the team who have no experience with server development at all.

Python is simple and efficient, and I have a deep understanding of it. There were a few programmers on the private cloud project at the time, but we had to serve a number of large enterprises and develop them in a customized way, thanks to Python, we were able to caller quickly. After the enterprise fast Plate was hung up, we started the light Office project and naturally used Python to build the original version.

Although Python is very powerful, but we also encountered some problems when using, mainly by the following aspects:

• Dynamic language

  Python is a dynamic language, not a strongly typed system. For a variable, we sometimes do not know what type it is, and then there may be a run-time error such as INT + string.

  In Python, you can allow a function with the same name to appear, and the latter will overwrite the previous function, and one of our system's most serious errors is because of this.

  As mentioned above, the static language can be compiled to help us detect, and do not have to wait until the runtime problems to know. While we have well-developed test cases, there are always cases where case is missing. So every time there was a run-time error, I thought about how good it would be if I could compile it.

• Performance

  This has always been a place where many people spit out python, but think about what Python was developed to solve? We just want to use him to high-performance server development above, in fact, it is a bit difficult.

  Python's Gil leads to the inability to really multi-threading, and everyone might say I'm not going to end up with multiple processes. However, if some computations involve multi-process interaction, the communication overhead between processes has to be considered.

  Stateless distributed processing is convenient for using multiple processes, such as handling HTTP requests, and we are mounting more than 200 Django servers behind Nginx to handle HTTP, but so many processes naturally lead to high overall machine load.

  But even if we use multiple Django processes to handle HTTP requests, Python still can't handle some of the extra requests. So we use Openresty to implement high-frequency HTTP requests using LUA. This in turn leads to the use of two development languages, and some logic has to write two different codes.

• Synchronous Network Model

  The Django Network is a synchronous block, that is, if we need to access a service outside, Django cannot handle any other logic while waiting for the result to return (except, of course, multithreading). If it takes a long time to access an external service, it means that our entire service is completely unavailable for almost a long time.

  In order to solve this problem, we can only continue to open the Django process, but also to ensure that all services can quickly handle the response, but think this is really a very unreliable thing.

• Asynchronous Network Model

  The Tornado network model is asynchronous, which means that it does not appear as Django does because the service cannot respond because the external service is unavailable. In other words, compared to Django, I am very fond of tornado, small and simple, before also wrote a few in-depth analysis of tornado article.

  Although Tornado is asynchronous, Python's MySQL library does not support asynchrony, which means that if we access the database in tornado, we may still face the fact that the entire service is not available due to database problems.

  In fact, the biggest problem of the asynchronous model is the fragmentation of the code logic, because it is triggered by the event, so we are all through callback related processing, so the code inside often appear to do one thing, pass a callback, and then callback inside and pass callback situation, The result is that the entire code logic is very confusing.

  Python does not have native coprocessor support, although it is possible to support the process by gevent,greenlet this patch, but after all, the Python source code is changed. In addition, the yield of Python can also be a simple co-simulation, but after all, can not cross the stack, the limitations are very large, do not know whether the 3.x version has been improved.

• Develop operations deployment

  When I first used the Python development project, I did not successfully install the package required for the project, the installation of the successful MySQL library has been a long time. Later, it was a colleague who packaged the entire Python directory for me so I could run the project normally. In other words, now that there is Docker, it is one thing that makes people happy.

  While deploying the Python service, we need to install a bunch of packages on the server, which is very troublesome, although it is possible to solve the deployment problem by puppet,salt these automation tools, but compared to the static compilation language only throw a binary file, it is much more convenient.

• Code out of control

  Python is very flexible and simple, the ability to write C dozens of lines of code can be done, Python line of code can not be solved. But it is too simple, but many students can not be in-depth thinking of the code, the entire architecture for careful consideration. Come a demand, PA PA, the keyboard knocked out the speed of implementation, the result is more and more chaotic code, resulting in the entire project code out of control.

  Although this also has our own reasons, such as the lack of good code review Mechanism, no good project specifications, but the personal feeling, if a programmer is not good coding training, Python is easy to write bad code, because it is too free.

  Of course, I'm not saying that Python cannot be used to develop large projects, watercress, Dropbox are good examples, but in our project, our Python code is out of control.

The above mentioned are the problems we encountered in the actual project using Python, although eventually resolved, but it makes me increasingly feel that with the increase in project complexity, the increase in traffic performance pressure, Python is not a good choice.

Why Choose Go

Now that you've finished Python, let's say why we chose go. In fact, in addition to Python, we also have other options, Java,php,lua (Openresty), but in the end we chose go.

Although both Java and PHP are the best programming languages (everyone is so competitive), I prefer a simpler language. While Openresty, though powerful, Lua is still a dynamic language, and will encounter some of the previous dynamic language problems. Finally, before Jinshan Xu Xiwei used the go, the former fast-plate architect Onion also used go, so we naturally chose go.

Go is not perfect, a pile of places worth our while.

• Error, well, if a student who has a neat language may be really sick of the grammar of Go, especially the last return value of the contract is error. This code often fills the project:

    if err := doA(); err != nil {      if _, err := doB(); err != nil {      }  }

  No wonder there's a stem for a demand, Java programmers write configuration, go programmers have written most of the code, but when the Java programmer finished writing, go programmer is still writing err != nil .

• Package management, go package management is too weak, only a go get, that is, if you accidentally updated an external library, it is likely to lead to the existing code compiled. Although there are many open source programs, such as GODEP and now out of the GB, but it is not official. It seems that Google is also using the vendor mechanism to manage third-party libraries. Hopefully the go 1.5 or later version will take care of the problem.

• Gc,java GC Development for 20 years, go just so little time, GC is definitely not perfect. So we still can't write the code as we want, or the GC may be the whole service under the large request volume. So sometimes, the use of object pool, memory pool must be used, although the code is ugly point, but somehow performance up.

• Generics, although Go has inteface, but the lack of generics will let us in the implementation of a function to write a lot of repetitive code, such as int32 and int64 type of sort, we have to write two sets of code, good redundancy. Go 1.4 with Go generate support, but this still needs to be based on the AST library to manually write related parser, the difficulty is quite large. Although there are many open-source generate implementations, it is not official at all.

Of course there are a lot of places worth to spit out, not listed, but go still has its advantages.

• Static language, strongly typed. Static compilation can help us to check out a lot of errors, the strong type of go and even perverted to not support implicit type conversion. Although writing code feels awkward, it reduces the likelihood of making a mistake.
• GOFMT, it's supposed to be the first one I know. The language of the Uniform Format Code tool is provided. With GOFMT, everyone's code is the same, there is no curly braces in the end or a new line of this kind of egg-like code style discussed. Because the code style is the same, it's easy to see the code of Go.
• Natural parallel support, because Goroutine and channel, with go write distributed applications, write concurrency program exception easy. Without the callback of the egg, the code logic is fragmented and the code logic is sequential.
• Performance, go performance may not be able to catch up with c,c++ and openresty, but it's also pretty tough. In our project, a go process is now deployed on a single machine, fully capable of doing what was done in the previous 200 Python processes, with lower CPU and mem usage.
• Operations deployment, directly compiled into binary, thrown onto the server, more than Python needs to install a bunch of environment that is simply too much. Of course, if there are CGO, we also need to throw the corresponding dynamic library to the past.
• Development efficiency, although go is a static language, but I personally feel that the development efficiency is really very high, intuition above the same as Python. For me personally, the best example is that I quickly developed a lot of open source components, such as Ledisdb,go-mysql, with go, and the first versions were done in a very short time. For our project, we also completed the first version of refactoring with go in one months, and released it.

Some go Tips in real projects

Up to now, almost all of our service-side projects have turned to go, and, of course, there have been some problems when it comes to the use of it, which is listed as experience sharing.

• GODEP, we use GODEP for third-party library management, but GODEP the biggest pit I've ever met is the build tag problem, and if a file has a build TAG,GODEP it's likely that it will be ignored.
• IO deadline, if you can handle themselves in the application layer processing, go deadline internal is a timer to control, but the timer internal use an array to implement the heap, global share a lock, if the large concurrency, and the number of timer too many, Timeout changes are too frequent and can easily cause performance problems.
• GC, this front also said, multi-use memory pool, object pool, in addition, I also found that if the object's life cycle is consistent with goroutine, the performance of the upgrade is good, also in the Go Group asked related issues, It may be speculated that some objects are actually allocated on the 8k stack of goroutine, so there is no additional GC to recycle.
• Go gob, if you want to do the RPC service, GOB is not a good choice, first of all, and Python pickle is not common, and then in order to do different system data incoming, any package must carry the type of details, size is too large. Go inside now there is not a set of official RPC program, GRPC seems to have the upper possible.

Summarize

Although I chose go now, it does not mean that I will not try other languages in the future. Language is not good or bad, can help me solve the problem is the language. But at least for a long time, I would use go to develop. Let ' Go!!!