Thoughts on the best practices of the go language in the read Production Environment

Recently I read an article about go product development best practices, go-in-procution. The author sums up a lot of practical experience in the process of using go for development, and many of them are actually used. In view of this, here is a simple post-reading experience, I will try to translate this article later. I will use soundcloud to refer to the original author later.

Development Environment

In soundcloud, each person uses an independent GOPATH and directly clones the code in the GOPATH according to the code path specified by go.

$ mkdir -p $GOPATH/src/github.com/soundcloud$ cd $GOPATH/src/github.com/soundcloud$ git clone git@github.com:soundcloud/roshi

For go, the general project management should be the following directory structure:

proj/    src/        modulea/            a.go        moudleb/            b.go        app/            main.go    pkg/    bin/

Then, we set the proj path in GOPATH to compile and run the program. But if we want to submit the code to github and allow other developers to use it, we cannot submit the entire proj, it hurts. Outside developers may reference this as follows:

import "github.com/yourname/proj/src/modulea"

However, in the code, we can directly:

import "github.com/yourname/proj/modulea"

If the outside developers need to remove the src reference method, they can only set GOPATH to the proj directory. If there is more import, it will crash.

I have been suffering from this problem for a long time. After finally reading the vitess code, I found that the above method is very good.

Project directory structure

If there are not many files in a project, simply put them in the main package and you do not need to split them into multiple packages. For example:

github.com/soundcloud/simple/    README.md    Makefile    main.go    main_test.go    support.go    support_test.go

If there is a public class library, it will be split into separate packages for processing.

Sometimes, a project may contain multiple binary applications. For example, a job may require a server, a worker, or a janitor. In this case, multiple subdirectories are created as different main packages and different binary applications are placed respectively. Use other sub-directories to implement common functions.

github.com/soundcloud/complex/README.mdMakefilecomplex-server/    main.go    main_test.go    handlers.go    handlers_test.gocomplex-worker/    main.go    main_test.go    process.go    process_test.goshared/    foo.go    foo_test.go    bar.go    bar_test.go

My practice is a little different. I mainly refer to vitess. I like to create a general cmd directory and then set different subdirectories in it, in this way, you do not need to guess whether the directory is a library or an application.

Code style

There is nothing to say about the code style. You can use gofmt to solve it directly. We usually agree that gofmt does not include any other parameters.

It is best to configure your editor to automatically perform gofmt processing when saving code.

Google recently released the go code specification, and soundcloud has made some improvements:

• To avoid the return value of the namefunction, unless it can clearly indicate the meaning.
• Use make and new as little as possible, unless necessary, or know the size to be allocated in advance.
• Use struct {} as the tag value, instead of bool or interface {}. For example, set is implemented using map [string] struct {} Instead of map [string] bool.

If a function has multiple parameters and the length of a single row is long, you need to split them. You do not need to use the java method:

// Don't do this.func process(dst io.Writer, readTimeout,    writeTimeout time.Duration, allowInvalid bool,    max int, src <-chan util.Job) {    // ...}

Instead, use:

func process(    dst io.Writer,    readTimeout, writeTimeout time.Duration,    allowInvalid bool,    max int,    src <-chan util.Job,) {    // ...}

Similarly, when constructing an object, it is best to input relevant parameters during initialization, instead of setting them later:

f := foo.New(foo.Config{        Site: "zombo.com",
        Out:  os.Stdout,
        Dest: conference.KeyPair{
            Key:   "gophercon",        Value: 2014,    },})// Don't do this.f := &Foo{} // or, even worse: new(Foo)f.Site = "zombo.com"f.Out = os.Stdoutf.Dest.Key = "gophercon"f.Dest.Value = 2014

If some variables can be obtained only after other operations, I think they can be set later.

Configuration

Soundcloud uses the flag package of go to pass configuration parameters, instead of passing configuration files or environment variables.

The flag configuration is defined in the main function, rather than in the global scope.

func main() {    var (        payload = flag.String("payload", "abc", "payload data")        delay   = flag.Duration("delay", 1*time.Second, "write delay")    )    flag.Parse()    // ...}

I reserve my opinion on using flag as the configuration parameter. If an application requires a large number of configuration parameters, it is quite painful to use the flag. At this time, it is better to use the configuration file. I personally prefer to use json as the configuration, for this reason.

Logs

Soundcloud uses go log, which also shows that their log does not require many other functions, such as log classification. For the log, I wrote one by referring to the log of python, here. This log supports the log Level and custom loghandler.

Soundcloud also mentioned the concept of telemetry. I really cannot translate it. According to my understanding, it may be the collection of program information, including response time, QPS, and memory running errors.

Generally, telemetry can be pushed or pulled in two ways.

The push mode actively sends information to a specific external system, while the PULL mode writes the information to a certain place, allowing the external system to obtain the data.

Both methods have different positioning methods. If you need to view data in a timely and intuitive manner, the push mode is a good choice, but this mode may occupy too much resources, CPU and bandwidth are greatly consumed, especially when the data volume is large.

Soundcloud looks like a pull model.

I strongly agree on this. We have a service that needs to send its information to a statistical platform for a total of subsequent information. At the beginning, we use the push mode to generate each record, we pushed it directly to the following statistics platform through http. Finally, as the pressure increases, the entire statistics platform was suspended and declined to provide services. Finally, we use the method of writing data locally and then pulling and sending data through another program.

Test

Soundcloud uses the testing package of go for testing, and then uses the flag Method for integration testing, as shown below:

// +build integrationvar fooAddr = flag.String(...)func TestToo(t *testing.T) {    f, err := foo.Connect(*fooAddr)    // ...}

Because go test also supports flag transfer similar to go build, it will synthesize a main package by default, and then perform flag parse processing in it.

I didn't use this method now. I wrote a global configuration directly in the test case to facilitate the go test... processing in the root directory. However, I think it is flexible to use the flag method, which may be considered later.

The testing package of go does not provide strong functions. For example, assert_equal is not provided, but we can solve it through reflect. DeepEqual.

Dependency Management

This is what I really want to solve. Now our code is very violent to use go get to solve the dependency problem. This is actually very risky. If a dependency package changes the interface, so we may have problems with go get.

Soundcloud manages dependencies in the vendor mode. In fact, it is very easy to copy all the dependent items to your project and use them as your own code. However, this requires regular maintenance of dependent package updates.

If an executable package is introduced, create a _ vendor folder under your project directory (in this way, tools related to go, such as go test, will ignore the contents of this folder ). Use _ vendor as a separate GOPATH. For example, copy github.com/user/depto the directory of _vendor/src/github.com/user/dep. Add _ vendor to your GOPATH as follows:

GO ?= goGOPATH := $(CURDIR)/_vendor:$(GOPATH)all: buildbuild:    $(GO) build

If a library is introduced, add it to the vendor directory and use vendor as the prefix of the package. For example, copy github.com/user/depto vendor/user/depand modify all importsentences.

Because we do not need to perform go get update on these introduced projects frequently, It is very worthwhile to do so most of the time.

I used a similar method at the beginning, but I did not call it vendor, but 3rd. Later I decided to change it to go get for convenience, although I knew this was a big risk. Maybe using godep in the future may be a good solution.

Build and deploy

Soundcloud directly uses go build to build the system during development, and then uses a Makefile to process the formal build.

Soundcloud is mainly used to deploy many stateless services. Similar to Heroku, soundcloud provides a simple method:

$ git push bazooka master$ bazooka scale -r <new> -n 4 ...$ # validate$ bazooka scale -r <old> -n 0 ...

In this regard, we directly use a simple Makefile to build the system, as shown below:

all: build build:    go install ${SRC}clean:    go clean -i ${SRC}test:    go test ${SRC} 

Application publishing uses the original scp method to restart the target machine. However, salt is being tested to release the application. For application startup and stop, we use the supervisor for management.

Summary

In general, this article explains in detail a lot of experience in using go for product development, hoping to help you.