9 Lessons to build teamwork projects with Go

This is a creation in Article, where the information may have evolved or changed.

Original: Go learn:learning as we Go
Author: Senior engineer of Peter Kelly,teamwork Desk
translation: Wei Sun
Zebian: The dawn of money, focus on architecture and algorithmic areas, seek reports or contributions please email qianshg@csdn.net, another "csdn Senior Architect Group ", there are many well-known internet companies, Daniel Architects, welcomed the architect plus qshuguang2008 application Group, notes name + company + position.

Summary: The teamwork team wrote nearly 200,000 lines of go code last year, building a bunch of small, fast HTTP services, and this article lists 9 lessons they have summed up.

Why Choose Go language? Go language, also known as Golang, is a static strong type, compiled, and hairstyles developed by Google, and has a garbage collection mechanism programming language, it runs very fast, but also has the following features: a first-class standard library, no inheritance, support multi-core At the same time, it also has legendary designers and extremely good community support, not to mention the authors of our web applications are exceptionally convenient, You can avoid the event loop and callback Goroutine-per-request set (each request processing requires a separate goroutine to be started). At present, the go language has become a popular choice for building systems, servers, and especially microservices.

As with other emerging languages or technologies, we have had a good period of groping in the early stages of experimentation. The go language does have its own style and usage, especially for developers who turn from object-oriented languages (such as Java) or scripting languages (such as Python). So we made a lot of mistakes, and in this article we want to share what we've got. If you use the Go language in a production environment, these questions are likely to come up, and hopefully this article will provide some help for beginners in the go language.

1. Revel is not a good choice

For the novice go language users who need to build a Web server, they may think that a suitable framework is needed at this time. There are advantages to using the MVC framework, mainly due to the set of project architectures and practices that are established by the practice precedence principle, which gives project consistency and lowers the threshold for cross-project development. But we found that self-configuration is more powerful than convention, especially that the go language has minimized the difficulty of writing Web applications, and many of our web applications are small services. The most important thing is that our application is not in accordance with Convention.

Revel is designed to try to introduce frameworks like play or rails into the go language, rather than using the power of Go and stdlib to build on it. According to the writers of the Go language:

Initially it was just an interesting project and I wanted to try to replicate the magical play-frame experience in the less magical go language.

To be fair, it makes sense for us to adopt the MVC framework in a new language-without having to argue about the architecture, and the new team is able to build content consistently. Before I used the go language, every web app I wrote had traces of the MVC framework. The use of ASP. NET MVC in C #, the use of SPRINGMVC in Java, the use of symfony in PHP, the use of cherrypy in Python, the use of Ror in Ruby, but finally we found that there is no need for a framework in the go language. The standard library HTTP package already contains the required content, generally as long as the multiplexer (such as MUX) to select the route, and then join the LIB to handle the middleware (such as Negroni) tasks (including authentication and login, etc.) is sufficient.

The standard library HTTP package design of Go makes this work very simple, and users will gradually discover that the power of Go is partly due to its toolchain and related tools--which contain a variety of powerful commands that can be run in code. However, in Revel, we cannot use these tools due to the setup of the project architecture, coupled with the lackpackage mainandfunc main() {}ingress (which are both customary and necessary go commands). In fact, Revel comes with its own command pack, mirroring some similarrunbuildcommands.

After using Revel, we:

• Unable to rungo build;
• Unable to rungo install;
• Unable to use race detector (–race);
• Inabilitygo-fuzzto use or other powerful tools that need to build go resources;
• Unable to use other middleware or routing;
• Although the thermal overload is concise, but slow, Revel uses a reflection mechanism (reflection) on the source, and the compilation time is increased by about 30% from the 1.4 version.go installThe package is not cached because it is not used;
• Since the compilation is slower in Go 1.5 and later, it cannot be migrated to a higher version, and in order to upgrade the kernel to version 1.6, we removed the revel;
• Revel put the test under/test dir, violating the go language_test.go's habit of packaging files with test files;
• To run the Revel test, you need to start the server and perform the integration test.

We found that many of Revel's ways are far from the build habits of the go language and have lost some of the powerful go toolset.

2. Use panics wisely

If you're a developer of the go language from Java or C #, there may be some less accustomed to error handling in the Go language (error handling). In the go language, functions can return multiple values, so it is typical to return an error when returning other values, and if everything works, the value returned by Resturnserror is nil (nil is the default value for reference types in the Go language).

func something () (thing string, err error) {
    s: = db.GetSomething ()
    if s == "" {
        return s, errors.New ("Nothing Found")
    }
    return s, nil
}
Since we wanted to create an error and handle it at a higher level in the call stack, we finally used panic.

s, err: = something ()
    if err! = nil {
    panic (err)
}
We were completely stunned: an error? God, run it!

But in Go, you will find that error is actually a return value, which is very common in function calls and response processing, and panic will slow down the performance of the application and cause a crash-similar to a crash when running abnormally. Why do you just do it because you need the function to return error? This is our lesson. Before the 1.6 version was released, the dump panic stack was also responsible for dumping all running Go programs, which made it very difficult to find the origin of the problem. We searched for a lot of unrelated content for a long time and tried in vain.

Even if there is a truly unrecoverable error, or a panic at runtime, you probably do n’t want the entire web server to crash, because it is also a middleware for many other services (your database also uses transaction mechanisms, right? ) So we learned how to deal with these panic: add filter in Revel to restore these panic, and also get the stack trace record in the log file and send it to Sentry, and then send us a warning through email and Teamwork Chat live chat , The API returns "500 internal server error" to the front end.

// PanicFilter wraps the action invocation in a protective defer blanket that
// recovers panics, logs everything, and returns 500.
func PanicFilter (rc * revel.Controller, fc [] revel.Filter) {
    defer func () {
        if err: = recover (); err! = nil {
            handleInvocationPanic (rc, err) // stack trace, logging. alerting
        }
    } ()
    fc [0] (rc, fc [1:])
}
3. Beware of reading from Request.Body more than once
After reading the content from http.Request.Body, its Body is evacuated, and then reading it again will return an empty body [] byte {}. This is because when reading a http.Request.Body data, the reader will stop at the end of the data, you must first reset it if you want to read it again. However, http.Request.Body is an io.ReadWriter, and does not provide a method such as Peek or Seek to solve this problem. One solution is to copy the Body into memory first, and then fill in the original content after reading it. If there are a large number of requests, this method is very expensive and can only be considered expedient.

Here is a short and complete code:

package main

import (
    "bytes"
    "fmt"
    "io / ioutil"
    "net / http"
)

func main () {
    r: = http.Request {}
    // Body is an io.ReadWriter so we wrap it up in a NopCloser to satisfy that interface
    r.Body = ioutil.NopCloser (bytes.NewBuffer ([] byte ("test")))

    s, _: = ioutil.ReadAll (r.Body)
    fmt.Println (string (s)) // prints "test"

    s, _ = ioutil.ReadAll (r.Body)
    fmt.Println (string (s)) // prints empty string!
}
This includes the code for copying and backfilling:

content, _: = ioutil.ReadAll (r.Body)
// Replace the body with a new io. ReadCloser that yields the same bytes
r.Body = ioutil.NopCloser (bytes.NewBuffer (content))
again, _ = ioutil.ReadAll (r.Body)
You can create some util functions:

func ReadNotDrain (r * http.Request) (content [] byte, err error) {
    content, err = ioutil.ReadAll (r.Body)
    r.Body = ioutil.NopCloser (bytes.NewBuffer (content))
    return
}
Instead of calling a method similar to ioutil.ReadAll:

content, err: = ReadNotDrain (& r)
Of course, now that you have replaced r.Body.Close () with no-op, no operation will be performed when Close is called in request.Body, which is how httputil.DumpRequest works.

4. Some continuously optimized libraries are helpful for writing SQL
In Teamwork Desk, the core function of providing web application services to users often involves MySQL, and we do not use stored procedures, so the data layer in Go contains some very complicated MySQL ... and the queries built by some code are complex The degree is comparable to the champion of the Olympic gymnastics competition. At the beginning, we used Gorm and its chainable API to build SQL. In Gorm, you can still use the original SQL and let it generate results based on your structure (but in practice, we have recently found that such operations are getting more and more Frequently, this means that we need to readjust the way we use Gorm to ensure the best way is found, or need to see more alternatives-but there is nothing to be afraid of!)

For some people, object-relational mapping (ORM) is very bad, it will make people lose control and understanding, and the possibility of optimizing queries. This idea is correct, but we only use Gorm as a query (understand The output part of the package), rather than completely used as an ORM. In this case, we can use its chainable API to construct queries as follows, and adjust the results according to the specific structure. Many of its functions are convenient for handwriting SQL in the code, and also support Preloading, Limits, Grouping, Associations, Raw SQL, Transactions and other operations. If you want to handwrite SQL code in Go language, then this method is worth a try.

var customer Customer
   query = db.
   Joins ("inner join tickets on tickets.customersId = customers.id").
   Where ("tickets.id =?", E.Id).
   Where ("tickets.state =?", "Active").
   Where ("customers.state =?", "Cork").
   Where ("customers.isPaid =?", False).
   First (& customer)
5. Pointers without pointers are meaningless
In fact, this refers specifically to slices. Did you use slices when passing values to functions? In Go, an array is also a numeric value. If you have a large number of arrays, you do n’t want to copy it every time you pass a value or assign it? That's right, the overhead of passing an array of memory is huge, but in Go, 99% of the time we deal with slices instead of arrays. In general, a slice can be used as a description of some fragments of an array (often all fragments), including a pointer to the beginning element of the array, the length and capacity of the slice.

Each part of the slice only needs 8 bytes, so no matter what the bottom layer is, the size of the array will not exceed 24 bytes.

We often send pointers to function slices, thinking they can save space.

t: = getTickets () // e.g. returns [] Tickets, a slice
ft: = filterTickets (& t)

func filterTickets (t * [] Tickets) [] Tickets {}
If there is a lot of data in t, we think that sending it to filterTicket can prevent the copying of large amounts of data in memory. Now with the understanding of slicing, we know that you can only send the slice value without worrying about memory.

t: = getTickets () // [] Tickets massive list of tickets, 20MB
ft: = filterTickets (t)

func filterTickets (t [] Tickets) [] Tickets {} // 24 bytes passed by value
Of course, not sending by reference also means that you will not make incorrect changes to the pointer, because the slice itself is a reference type.

6. Naked returns loses readability and makes the code more difficult to read (in larger functions).
In the Go language, "Naked returns" refers to the return from a function without specifying the return content.

Go language can have named return values, such as func add (a, b int) (total int) {}, I can use the function just returned to perform the return without returning all the content (use return instead of return all). Naked Returns is very useful and compact in small functions.

func findTickets () (tickets [] Ticket, countActive int64, err error) {
    tickets, countActive = db.GetTickets ()
    if tickets == 0 {
        err = errors.New ("no tickets found!")
    }
    return
}
Obviously, if the ticket is not found, it returns 0, 0, error; if the ticket is found, it returns a format like 120, 80, nil, the specific value depends on the ticket count. The key is: if the return value is named in the function signature, you can use return (naked return), and when the call returns, it will also return the state of each named return value.

However, we have some large functions that are a little bulky. In the function, the naked returns of any length that need to turn the page will greatly affect the readability, and it is easy to cause subtle bugs. Especially if there are multiple return points, never use naked returns or large functions.

Below is an example:

func findTickets () (tickets [] Ticket, countActive int64, err error) {
    tickets, countActive: = db.GetTickets ()
    if tickets == 0 {
        err = errors.New ("no tickets found!")
    } else {
        tickets + = addClosed ()
        // return, hmmm ... okay, I might know what this is
        return
    }
    .
    .
    .
    // lots more code
    .
    .
    .
    if countActive> 0 {
        countActive-closedToday ()
        // have to scroll back up now just to be sure ...
        return
    }
    .
    .
    .
    // Okay, by now I definitely can't remember what I was returning or what values they might have
    return
}
7. Beware of scope and abbreviations
In the Go language, if you use the same abbreviated name in different block areas: = to declare variables, due to the existence of scope, some subtle bugs will occur, which we call shadowing.

func findTickets () (tickets [] Ticket, countActive int64) {
    tickets, countActive: = db.GetTickets () // 10 tickets returned, 3 active
    if countActive> 0 {
        // oops, tickets redeclared and used just in this block
        tickets, err: = removeClosed () // 6 tickets left after removing closed
        if err! = nil {
            // Argh! We used the variables here for logging !, if we didn't we would
            // have received a compile-time error at least for unused variables.
            log.Printf ("could not remove closed% s, ticket count% d", err.Error (), len (tickets))
        }
    }
    return // this will return 10 tickets o_O
}
Specifically: = The problem of declaration and allocation of abbreviated variables. Generally speaking, it will be compiled if a new variable is used on the left: =, but it is also valid if other new variables appear on the left. In the above example, err is a new variable, because it has been declared in the parameters returned by the function, you think the ticket will be automatically overwritten. But this is not the case. Because of the scope of the block, after the new ticket variable is declared and assigned, once the block is closed, its scope will be lost. In order to solve this problem, we only need to declare that the variable err is outside the block, and then use = instead of: =, an excellent editor (such as Emacs or Sublime with the Go plugin can solve this shadowing problem).

func findTickets () (tickets [] Ticket, countActive int64) {
    var err error
    tickets, countActive: = db.GetTickets () // 10 tickets returned, 3 active
    if countActive> 0 {
        tickets, err = removeClosed () // 6 tickets left after removing closed
        if err! = nil {
            log.Printf ("could not remove closed% s, ticket count% d", err.Error (), len (tickets))
        }
    }
    return // this will return 6 tickets
}
8. Mapping and random crash
During concurrent access, the mapping is not secure. We have seen this situation: the mapping is used as an application-level variable for the entire life cycle of the application. In our application, this mapping is used to collect statistics for each controller. Of course, in the Go language, each http request is Own goroutine.

You can guess what will happen below. In fact, different goroutines will try to access the map at the same time, or it may be read or written, which may cause panic and cause the application to crash (we used the upstart script in Ubuntu, Restart the application when the process is stopped, at least to ensure that the application is considered "online"). The interesting thing is that this situation occurs randomly. Before version 1.6, trying to find out the reason for panic like this was a little hard, because the stack dump contains all the goroutines in the running state, which leads to the need to filter a large number of logs.

During concurrent access, the Go team did consider the security of mapping, but eventually gave up, because in most cases this method will cause unnecessary overhead, as explained in the FAQ of golang.org:

After a long discussion, we decided that when using mapping, it is generally not necessary to perform secure access from multiple goroutines. When secure access is indeed required, the mapping is likely to belong to a larger data architecture or calculation that has been synchronized. Therefore, if all mapping operations are required to require a mutex, it will slow down most programs, but the effect is minimal. Since uncontrolled mapping access can crash the program, making this decision is not easy.

Our code looks like this:

package stats

var Requests map [* revel.Controller] * RequestLog
var RequestLogs map [string] * PathLog
We have modified it to use stdlib's synchronous packet: embed read / write mutexes in the structure of the package map. We added some helpers to this structure: Add and Get methods:

var Requests ConcurrentRequestLogMap

// init is run for each package when the app first runs
func init () {
    Requests = ConcurrentRequestLogMap {items: make (map [interface {}] * RequestLog)}
}

type ConcurrentRequestLogMap struct {
    sync.RWMutex // We embed the sync primitive, a reader / writer Mutex
    items map [interface {}] * RequestLog
}

func (m * ConcurrentRequestLogMap) Add (k interface {}, v * RequestLog) {
    m.Lock () // Here we can take a write lock
    m.items [k] = v
    m.Unlock ()
}

func (m * ConcurrentRequestLogMap) Get (k interface {}) (* RequestLog, bool) {
    m.RLock () // And here we can take a read lock
    v, ok: = m.items [k]
    m.RUnlock ()

    return v, ok
}
Now it will never collapse again.

9. Use of Vendor
Well, although it is difficult to tell, we just made this mistake, and the blame is very big-when deploying the code to the production environment, we did not use the vendor.

To explain briefly, in Go language, we get the dependencies by running go get. / ... from the root directory of the project. Each dependency needs to be pulled from the HEAD of the main server. Obviously this situation is very bad Unless the exact version of the dependency is saved on the server of $ GOPATH, and it has not been updated (nor rebuilt or run a new server), if the change is unavoidable, you will lose control of the code running in the production environment. In the Go 1.4 version, we used Godeps and its GOPATH to execute the vendor; in the 1.5 version, we used the GO15VENDOREXPERIMENT environment variable; in the 1.6 version, finally no tools are needed-/ vendor in the project root directory can be automatically identified It is a dependent storage location. You can choose one of the different vendor tools to track the version number, making it easier to add and update dependencies (remove .git, update list, etc.).

I have learned a lot, but I have never stopped learning
The above lists only a small part of our initial mistakes and lessons learned. We are just a small team of 5 developers who created the Teamwork Desk. Although we gained a lot in the Go language last year, there are a large number of excellent features swarming. This year we will attend various conferences on the Go language, including the GopherCon conference in Denver; I also discussed the use of Go at a local developer meeting in Cork.

We will continue to release open source tools related to the Go language, and are committed to giving back to existing libraries. At present, we have provided some small projects (see list), and the Pull Requests have been adopted by Stripe, Revel, and some other open source Go projects.

s3pp
stripehooks
tnef parser