Go language Practice: from beginners to online real-life small-scale services encounter those pits

This is a creation in Article, where the information may have evolved or changed.

Original: Go learn:learning as we Go
Author: Senior engineer of Peter Kelly,teamwork Desk
translation: Wei Sun
Zebian: The dawn of money, focus on architecture and algorithmic areas, seek reports or contributions please email qianshg@csdn.net, another "csdn Senior Architect Group ", there are many well-known internet companies, Daniel Architects, welcomed the architect plus qshuguang2008 application Group, notes name + company + position.

Summary: The teamwork team wrote nearly 200,000 lines of go code last year, building a bunch of small, fast HTTP services, and this article lists 9 lessons they have summed up.

Why Choose Go language? Go language, also known as Golang, is a static strong type, compiled, and hairstyles developed by Google, and has a garbage collection mechanism programming language, it runs very fast, but also has the following features: a first-class standard library, no inheritance, support multi-core At the same time, it also has legendary designers and extremely good community support, not to mention the authors of our web applications are exceptionally convenient, You can avoid the event loop and callback Goroutine-per-request set (each request processing requires a separate goroutine to be started). At present, the go language has become a popular choice for building systems, servers, and especially microservices.

As with other emerging languages or technologies, we have had a good period of groping in the early stages of experimentation. The go language does have its own style and usage, especially for developers who turn from object-oriented languages (such as Java) or scripting languages (such as Python). So we made a lot of mistakes, and in this article we want to share what we've got. If you use the Go language in a production environment, these questions are likely to come up, and hopefully this article will provide some help for beginners in the go language.

1. Revel is not a good choice

For the novice go language users who need to build a Web server, they may think that a suitable framework is needed at this time. There are advantages to using the MVC framework, mainly due to the set of project architectures and practices that are established by the practice precedence principle, which gives project consistency and lowers the threshold for cross-project development. But we found that self-configuration is more powerful than convention, especially that the go language has minimized the difficulty of writing Web applications, and many of our web applications are small services. The most important thing is that our application is not in accordance with Convention.

Revel is designed to try to introduce frameworks like play or rails into the go language, rather than using the power of Go and stdlib to build on it. According to the writers of the Go language:

Initially it was just an interesting project and I wanted to try to replicate the magical play-frame experience in the less magical go language.

To be fair, it makes sense for us to adopt the MVC framework in a new language-without having to argue about the architecture, and the new team is able to build content consistently. Before I used the go language, every web app I wrote had traces of the MVC framework. The use of ASP. NET MVC in C #, the use of SPRINGMVC in Java, the use of symfony in PHP, the use of cherrypy in Python, the use of Ror in Ruby, but finally we found that there is no need for a framework in the go language. The standard library HTTP package already contains the required content, generally as long as the multiplexer (such as MUX) to select the route, and then join the LIB to handle the middleware (such as Negroni) tasks (including authentication and login, etc.) is sufficient.

The standard library HTTP package design of Go makes this work very simple, and users will gradually discover that the power of Go is partly due to its toolchain and related tools--which contain a variety of powerful commands that can be run in code. However, in Revel, we cannot use these tools due to the setup of the project architecture, coupled with the lack package main and func main() {} ingress (which are both customary and necessary go commands). In fact, Revel comes with its own command pack, mirroring some similar run build commands.

After using Revel, we:

• Unable to run go build ;
• Unable to run go install ;
• Unable to use race detector (–race);
• Inability go-fuzz to use or other powerful tools that need to build go resources;
• Unable to use other middleware or routing;
• Although the thermal overload is concise, but slow, Revel uses a reflection mechanism (reflection) on the source, and the compilation time is increased by about 30% from the 1.4 version. go installThe package is not cached because it is not used;
• Since the compilation is slower in Go 1.5 and later, it cannot be migrated to a higher version, and in order to upgrade the kernel to version 1.6, we removed the revel;
• Revel put the test under/test dir, violating the go language _test.go 's habit of packaging files with test files;
• To run the Revel test, you need to start the server and perform the integration test.

We found that many of Revel's ways are far from the build habits of the go language and have lost some of the powerful go toolset.

2. Use panics wisely

If you're a developer of the go language from Java or C #, there may be some less accustomed to error handling in the Go language (error handling). In the go language, functions can return multiple values, so it is typical to return an error when returning other values, and if everything works, the value returned by Resturnserror is nil (nil is the default value for reference types in the Go language).

stringerror) {      s := db.GetSomething()    if"" {        return s, errors.New("Nothing Found")    }    returnnil}

Because we wanted to create an error and process it at the higher level of the call stack, we ended up using panic.

err := something()      iferr != nil {    panic(err)}

As a result we were completely shocked: an error? Oh, God, run it!

But in go, you'll find that error is also a return value, which is common in function calls and response processing, while panic slows down the performance of your application and causes crashes-like crashes when running an exception. Why do you do this just because you need the function to return an error? This is our lesson. Before the release of version 1.6, the dump panic Stack was also responsible for dumping all running go programs, which made it very difficult to find the origin of the problem, and we looked for a long, wasted effort on a whole bunch of unrelated content.

Even if you have a truly unrecoverable error, or run-time panic, you probably don't want the entire Web server to crash because it's also a middleware for many other services (your database also uses transactional mechanisms, right?). So we learned how to deal with these panic: adding filter in Revel would allow these panic to recover, get stack trace records from log files and send them to sentry, and send us a warning via email and teamwork chat live chat. The API returns "500 Internal server error" to the front end.

// PanicFilter wraps the action invocation in a protective defer blanket that// recovers panics, logs everything, and returns 500.func PanicFilter(rc *revel.Controller, fc []revel.Filter) {      deferfunc() {        ifrecovernil {            // stack trace, logging. alerting                    }    }()    fc[0](rc, fc[1:])}

3. Beware of reading from Request.body more than once

http.Request.Bodyafter reading the contents, the body is evacuated, and then read again will return to the empty body []byte{} . This is because when reading one http.Request.Body data, the reader stops at the end of the data and wants to read again must be reset first. However, http.Request.Body it is one io.ReadWriter that does not provide a way to solve this problem, such as peek or seek. One solution is to first copy the body into memory, and then fill in the original content back. If there is a large number of request, the cost of this approach is very high, only a stopgap measure.

Here is a short and complete code:

Package Mainimport ("bytes"    "FMT"    "Io/ioutil"    "Net/http") Func Main () {r: = http. Request{}//Body is an IO. ReadwriterSo we wrap it upinchA nopcloser to satisfy that interface R. Body= Ioutil. Nopcloser(bytes. Newbuffer([]byte ("Test"))) S, _: = Ioutil. ReadAll(r. Body) FMT. Println(String (s))//prints"Test"S, _ = Ioutil. ReadAll(r. Body) FMT. Println(String (s))//Prints Empty string! }

Here are the code for copying and backfilling:

content, _ := ioutil.ReadAll(r.Body)  // Replace the body with a new io.ReadCloser that yields the same bytesr.Body = ioutil.NopCloser(bytes.NewBuffer(content))  again, _ = ioutil.ReadAll(r.Body)  

You can create some util functions:

func ReadNotDrain(r *http.Request) (content []byte, err error) {      content, err = ioutil.ReadAll(r.Body)    r.Body = ioutil.NopCloser(bytes.NewBuffer(content))     return}

In a similar way as an alternative call ioutil.ReadAll :

content, err := ReadNotDrain(&r)  

Now, of course, you've replaced R with No-op. Body.close (), at request. When Close is called in the body, no action is taken, which is also httputil. The way dumprequest works.

4. Some continuously optimized libraries contribute to the writing of SQL

In teamwork Desk, the core functionality of providing web App services to users often involves MySQL, and we don't use stored programs, so the data layer in go contains some very complex mysql ... And some code constructs a query that is complex enough to rival the Olympic Gymnastics Championship. In the beginning, we built SQL with Gorm and its chain API, still using the raw SQL in Gorm, and let it produce results based on your structure (but in practice, we've recently found that this kind of operation is getting more frequent, which means we need to readjust the way we use Gorm to make sure we find the best way , or need to see more alternatives-but there's nothing to be afraid of! ）

For some people, the idea of object-relational mapping (ORM) is very bad, it can make people lose control and understanding, and the possibility of optimizing queries, but we just use Gorm as a way to build a query (the part that understands its output), rather than using it as an ORM. In this case, we can use its chain API as follows to build the query and adjust the results based on the specific structure. Many of its features make it easy to write SQL in code, and also support preloading, Limits, Grouping, associations, Raw SQL, transactions, and so on, if you're writing SQL code in the Go language, Then this method is worth a try.

varCustomer Customer Query=Db.Joins ("INNER JOIN tickets on tickets.customersid = customers.id").   Where("tickets.id =?"E.ID).   Where("tickets.state =?","Active").   Where("customers.state =?","Cork").   Where("Customers.ispaid =?",false).First (&Customer

5. No pointing pointer is meaningless

In fact, this is specifically a slice (slice). Are you using a slice when passing a value to a function? In the go language, arrays are numeric values, and if you have a large number of arrays, you don't want to copy them every time you send or distribute them. Yes, the overhead of having memory pass arrays is huge, but in the go language, we're dealing with slices instead of arrays in 99% of the time. In general, slices can be described as part of an array fragment (often all fragments), containing pointers to the starting elements of the array, the length and capacity of the slices.

Each section of a slice requires only 8 bytes, so no matter what the underlying is, the number of arrays will not exceed 24 bytes.

We often send pointers to function slices in the hope of saving space.

:=// e.g. returns []Tickets, a slice  ft:= filterTickets(&*[]Tickets) []Tickets {}  

If t there is a lot of data in it, we thought we would send it filterTicket to copy work that would block the execution of large amounts of data in memory. Now, with the understanding of slices, we know that you can send only the slice values without worrying about the memory problem.

of tickets, 20MB  ft := filterTickets(t)funcby value  

Of course, not sending by reference also means that you don't point the pointer at the wrong change, because the slice itself is a reference type.

6. Naked returns will lose readability and make the code more difficult to read (in larger functions).

In the Go language, "Naked returns" refers to a return from a function that does not explicitly indicate what is returned.

The go language can have a named return value, for example func add(a, b int) (total int) {} , I can use the function I just returned to perform a return without having to return the entire content (instead of using it return return all ). In small functions, Naked returns is very useful and concise.

errerror) {      tickets, countActive = db.GetTickets()    if0 {        err = errors.New("no tickets found!")    }    return}

It is obvious that if ticket is not found, it is returned, 0, 0, error if ticket is found, the format is returned, 120, 80, nil depending on the count of ticket. The key is that if you name the return value in the function signature, you can use return (naked return), and when the call returns, the state of each named return value is returned.

However, we have some large functions that are large and bulky. In the function, any length of naked returns that need to be paged will greatly affect readability and can easily cause subtle and imperceptible bugs. Especially if there are multiple return points, don't use naked returns or large functions.

Here is an example:

funcFindtickets (Tickets []ticket, countactiveInt64, err Error) {tickets, countactive: = db. Gettickets ()ifTickets = =0{err = errors. New ("No tickets found!")    }Else{Tickets + = addclosed ()//Return, Hmmm...okay, I might know what's this is        return}    .    . .//Lots more code.    . .ifCountactive >0{Countactive-closedtoday ()// has to scroll back up now just to be sure ...        return}    .    . .//Okay, by now I definitely can ' t remember what I am returning or what values they might has    return}

7. Beware of scopes and thumbnail declarations

In the go language, if you use the same thumbnail name to declare variables in different chunks, there := are some subtle bugs that we call shadowing because of scope.

funcFindtickets (Tickets []ticket, countactiveInt64) {tickets, countactive: = db. Gettickets ()//Tickets returned, 3 active    ifCountactive >0{//Oops, tickets redeclared and used just in this blockTickets, err: = Removeclosed ()//6 tickets left after removing closed        ifErr! =Nil{//argh! We used the variables here for logging!, if we didn ' t we would            //has received a compile-time error at least for unused variables.Log. Printf ("Could not remove closed%s, ticket count%d"Err. Error (),Len(tickets)) }    }return //This would return ten tickets o_o}

In particular, := the issue of the Declaration and allocation of the abbreviation variable, generally, if the new variable is used on the left, it will compile := , but it is also valid if there are other new variables on the left. In the example above, err it is a new variable because it has been declared in the parameters returned by the function and you think ticket it will be overwritten automatically. This is not the case, however, because of the existence of the chunk scope, when the new ticket variable is declared and allocated, its scope is lost once the chunk is closed. To solve this problem, we just need to declare that the variable err is outside the chunk, and then replace it with an = := excellent editor (such as Emacs or sublime that joins the go plugin to solve this shadowing problem).

funcint64) {      var err error    // 10 tickets returned, 3 active    if countActive > 0 {        // 6 tickets left after removing closed        ifnil {            log.Printf("could not remove closed %s, ticket count %d"len(tickets))        }    }    return// this will return 6 tickets}

8. Mapping and Random crashes

Mappings are not secure during concurrent access. We have had this situation: mapping as an application-level variable throughout the application's lifecycle, in our application, this mapping is used to collect each controller statistic, and of course every HTTP request in the Go language is its own goroutine.

You can guess what happens next, in fact different goroutine will try to access the mapping at the same time, or it could be a read or a write, which could cause the application to crash (we used the upstart script in Ubuntu to restart the app when the process stopped. At least ensure that the application is "online"). Interestingly: This happens randomly, before the 1.6 release, it's a bit of a struggle to figure out why there's a panic like this, because the stack dump contains all the goroutine in the running state, causing us to filter a lot of logs.

In concurrent access, the Go team did consider the security of the mappings, but eventually gave up because in most cases this would cause unnecessary overhead, as explained in the golang.org FAQ:

After a long discussion, we decided that when mapping is used, there is generally no need to perform secure access from multiple goroutine. When secure access is really required, the mapping is likely to belong to a larger data schema or computation that has already been synchronized. Therefore, if you require that all mapping operations require mutexes, you will slow down most programs, but with few effects. It is not easy to make this decision because the program crashes due to uncontrolled mapping access.

Our code looks like this:

packagemap[*revel.Controller]*RequestLog  map[string]*PathLog  

We modified it to use Stdlib's synchronous packet: Embed read/write mutexes in the structure of the encapsulated map. We have added some helper:add and get methods for this structure:

varRequests Concurrentrequestlogmap//init is run for each package when the app first runsfuncInit () {requests = Concurrentrequestlogmap{items: Make(Map[Interface{}]*requestlog)}}typeConcurrentrequestlogmapstruct{Sync. Rwmutex//We embed the Sync primitive, a reader/writer MutexItemsMap[Interface{}]*requestlog}func(M *concurrentrequestlogmap) ADD (kInterface{}, v *requestlog) {M.lock ()//Here we can take a write lockM.items[k] = v m.unlock ()}func(M *concurrentrequestlogmap) Get (kInterface{}) (*requestlog,BOOL) {M.rlock ()//And here we can take a read lockV, OK: = M.items[k] M.runlock ()returnV, OK}

It's not going to break anymore.

9. Use of Vendor

Well, it's hard to be ashamed, but we just made this mistake and it's a big offense-we didn't use vendor when we deployed the code to the production environment.

To explain briefly, in the go language, we get dependencies by running from the project root, and go get ./... each dependency needs to be pulled from the head of the master server, which is obviously very bad, unless the exact version of the dependency is saved on the $gopath server. And never make updates (or rebuild or run a new server), and if the changes are unavoidable, you will lose control of the code running in the production environment. In the Go 1.4 release, we used Godeps and its gopath to perform vendor, and in version 1.5 we used the GO15VENDOREXPERIMENT environment variable, and by the 1.6 version, we finally didn't need the tools--the /vendor project root directory Can be automatically identified as a dependent storage location. You can select one of the different vendor tools to track the version number, making it easier to add and update dependencies (remove. Git, update inventory, etc.).

A lot, but the learning

The above lists only a small part of our initial mistakes and the experience we have received. We're just a small team of 5 developers who created teamwork Desk, and despite the many things we've done in the go language last year, there's a huge influx of great features. This year we will attend various conferences on the go language, including the Gophercon conference in Denver, and I also discussed the use of go at the local developer's party in Cork.

We will continue to publish the go language-related open source tools and are committed to giving back to existing libraries. Now that we have provided some small projects (see list), the pull request was also adopted by stripe, revel, and some other open source go projects.

• s3pp
• Stripehooks
• TNEF parser