Best practices for Go language in a product environment

This is a creation in Article, where the information may have evolved or changed.

At SoundCloud, we built a product API for our customers. Or, our main website, mobile client and mobile app are the first customers of the API. Behind the API is a domain-based service: SoundCloud basically operates in the form of service-oriented architecture.

We are also a multilingual organization because we use a lot of languages. Also many parts of these services (and infrastructure support) are developed using Golang. In fact, we are all users of early Golang: At present, we have been using Golang in our products for 2.5 of the time. Related projects include:

• Bazooka, our internal service platform; Product ideas are very similar to Keroku or Flynn.
• Our peripheral transport layer uses generic Nginx, Haproxy, and so on, but they work with the Golang service.
• Our audio is stored on AWS S3, but uploading, transcoding, and generating links need to be handled Golang services.
• The search employs Elasticsearch to detect complex machine learning models, but they are integrated with the infrastructure developed by Golang.
• Prometheus, an early stage telemetry system is purely golang developed.
• Currently, stream processing takes Cassandra, but we are going to (almost) completely use Golang instead.
• We are also experimenting with HTTP streaming live services developed with Golnag.
• Many other small-product-oriented services.

These projects are developed by about six teams, including more than 10 SoundCloud, most of whom use Golang full-time. After all, at this time, these projects and such a mix of engineers, we have gradually formed the use of Golang in the product of the best practice method. These lessons will help other organizations that are starting to invest heavily in Golang.

Development environment

In our notebooks, we have set a single, global Gopath. Personally, I like to use $home, but many other people use a subdirectory under $home. We clone the warehouse into the relative path of the Gopath and then we can work directly. That

Many of us have been fighting the conventions in the early days to keep our own unique code-organizing approach. In fact, it doesn't deserve to be so troublesome at all.

For editors, many users use vim and a variety of plugins. (The vim-go I use is good.) There are also many people, including myself, that combine gosublime using sublime Text. There are also a few people using Emacs, but no one uses the IDE. I'm not sure this is the best practice, but it's interesting to mark out.

Library structure

Our best practice is to make sure everything is simple. Many service sources are packaged in the main package.

Our search scheduler, for example, is still the case two years later. Do not create a new structure until you determine the need.

Maybe at some point you need to create a new support package. Use subdirectories in your main library and import them with the full qualified name. If the package has only one file or one structure, then it certainly does not need to be split up.

Sometimes a repository needs to contain multiple binaries; For example, this task requires a service, a worker process, or a monitoring. In this case, place each binary file in a separate subdirectory of the specific main package and use the other subdirectories (or packages) to implement the shared functionality.

Note that you do not introduce the ASRC directory. Do not include the SRC directory in the repository or add it to the Gopath because of the vendor subdirectory exception (more on this section).

Format and Style

In general, first configure your editor to save the code to go FMT (or goimports), using the default parameters. This means using the tab indent and aligning with spaces. Malformed code will not be committed.

The past style guides are very broad, but Google recently released their Code review documentation, which is almost the convention we should abide by. Therefore, we use it.

We actually pushed it a little bit:

Avoid naming return parameters unless they can explicitly and significantly improve transparency.

Avoid using make and new unless they are necessary (new (int), or make (Chan int)), or we can know in advance the size of the items to be allocated (made (map[int]string,n), or made ([]int,0,256)).

Use struct{} as the tag value instead of Boolean or interface {}. For example, the collection is map[string]struct{}; The channel is Chan struct{}. It clearly indicates a clear lack of information.

It is also good to break long lines of parameters. It's more like the Java style:


This is better:


When constructing an object, it is also divided into multiple lines:


In addition, when assigning a new object, passing the member value in the initialization section (as above) is better than setting it later.

Configuration

We tried to pass the configuration to the go program in several ways: parsing the configuration file, using the OS. Getenv directly from the environment to extract the configuration, a variety of value-added flag resolution package. Finally, the most economic principle is the ordinary package flag, its strict type and simple semantics for everything we need is absolutely adequate and good enough.

We primarily deploy 12-factor applications, and 12-factor applications pass the configuration through the environment. But even then, we use a boot script to convert the environment variable to flags. Flags serves as a clear and full-text surface area between the program and its operating environment. They are invaluable for understanding and operating procedures.

A good habit about flags is to define them in your main function. This prevents you from arbitrarily using them as global variables in your code, which allows you to strictly adhere to dependency injection for easy testing.


Logs and Telemetry

We've tried several log frameworks that provide features like log level, debug, route output, custom formatting, and more. Finally we select the package log. Because we only record actionable information. This means that serious, panic-level errors, or structured data that need to be handled manually are consumed by other machines. For example, the search forwarder sends every request that it uses context-sensitive processing, so our analytics workflow can see people in New Zealand often search for Lorde, or whatever.

We take into account the telemetry, any other amount released during a run: Request response time, QPS, run error, queue depth, and so on. And telemetry basically consists of two modes: Push and pull.

Push means releasing the indicator to a known system. such as graphite, STATSD, and Airbrake

Pull means exposing the indicator at some known locations and allowing known systems to erase them. For example, Expvar and Prometheus (and perhaps others)

Of course, both ways have their own existence. Push is intuitive and simple when you start to use it. But the growth of push indicators is counterintuitive: the bigger you get, the higher the cost. We used to find that pull is the only model at this scale on a particular size infrastructure. There are also many values that can reflect a running system. Therefore, the best practice is: Expvar or similar style.

Testing and validation

During the course of the year we tried a lot of test libraries and frameworks, but soon gave up most of them, and today all of our tests are tested by data-driven (table-driven) tests with normal packages. We do not have a strong or explicit complaint about the test/inspection package, and in addition, they do not provide great value at all. One thing is helpful: reflect. Deepequal makes it easier to compare any value (for example, expected to got).

Package testing is for unit testing, and it can be a bit cumbersome for integration testing. The external services that run depend on your integration environment, but we find a good way to integrate them. Write a integration_test.go and give it a integration build tag. Define (global) flags, such as service addresses and connection strings, using them in your tests.

Go test and go build tag, so you can call go test-tags=integration. It also synthesizes the flag. The Parse package is main, so any declared and visible flags will be processed and provided to you for testing.

By validating, I mean static code validation. Fortunately, Go has some very good tools. I found it useful to consider the phase of writing code when considering which tool to use.

Use this when you do this.
Save Go FMT (or goimports)
Build go vet,golint, or go test
Deploy Go test-tags=integration

Episode
So far, nothing is too crazy. As a survey to compile this list, let me notice just how ... The conclusion how uninteresting. It's a dull one. I would like to emphasize that these very lightweight, purely standard library conventions can really be extended to large groups of developers and diverse project ecosystems. You will never need your own error-checking framework, or test libraries, just because your codebase has exceeded a certain size, or just because it may grow more than a certain number of rows. You really don't need it. The standard syntax and usage is still elegant when the code is large.

Dependency Management
The State of dependency management is a hot issue in the Go ecosystem, and we have not yet thought of the perfect solution. However, we have chosen a compromise that seems to be a good one.

How important is your project? Your dependency management solution is ...
Well ... go get-d, and then pray!
Very good. Vendoring

(It is worth noting that we have a shocking number of long-term product services that still depend on the first option. However, because we generally do not use too many third-party code, and the main problem is usually detected during the compile phase, we are lucky to circumvent this problem.)

Vendoring means that copies are dependent on the project code base and are then used at compile time. Depending on what you download, here are two best practices for vendoring.

Download the Vendor directory name process
Binary _vendor plus gopath prefix compilation
Library Vendor overriding import statements

If you download binary, create a _vendor subdirectory at the root of the codebase. (with an underscore so that the go tool ignores it when it is processed, such as go test./...) Treat it as if it were treated like a gopath; For example, copy this dependency github.com/user/dep to _VENDOR/SRC/GITHUB.COM/USER/DEP. Then, write a so-called Divine compilation process that adds _vendor to the gopath that might exist. (Remember: Gopath is actually a list of paths, and when the Go tool processes import, it searches the list sequentially.) For example, you might have a top-level makefile file that looks like this:


If you are downloading a class library, create a vendor subdirectory on your root repository. Handling this is like adding a prefix to the package directory. For example, a copy of the project from GITHUB.COM/USER/DEP is put to VENDOR/USER/DEP. After that, rewrite all of your introduction (import), and their interrelationships. This is painful, and when the rest of the content needs go get compatibility, the most effective way to look at it is to make sure that it is actually rebuilt (actually-reproducible build). It is important to note that we seldom download the class library in practice, so this approach is cumbersome and effective.

How to copy a dependency to your own repository in practice is another hot topic. The simplest way to do this is to manually copy files from one clone, which is probably the best answer if you don't care about the upstream department's push. Some people use git submodules, but we find that they are very counterintuitive and difficult to manage (and for many people, this is documented). We have been very successful with the GIT subdirectory, and he works like a submodule. There are also a number of tools that are used to automate this work. Now, it looks like GODEP development is very positive, and it's worth studying.

Build and deploy

It's tricky to build and deploy, so it's tightly coupled to your operating environment. I want to describe our scenario because I think it is a good model, but it may not be applied directly to your organization.

As far as construction is concerned, we usually develop directly using go build, and a Makefile is used to trim the official build. This is mainly because we are familiar with multiple languages, and our tools need to do a minimal functional collection (least common multiple). Also, our build system starts with an empty environment and requires a compiler (Makefile file is hard to see!). ）。

For deployment, the greatest attraction to us is the state of being stateless.

Pattern sample Model Deployment name Deployment form
Stateless Request router 12-factor Scaling Containers
Stateful Redis none,really provisioningcontainers?

We primarily deploy stateless services in a manner similar to Heroku.

Conclusion
I intentionally made this a report of experience from a large organization running go in a relatively long production environment. Although these are all based on opinions, they are still only views, so please hold the reservation. That said, the biggest advantage of Go is its simple structure. The ultimate best practice is to embrace simplicity rather than try to bypass it.