Those problems when writing the Go Code [2nd issue]

This is a creation in Article, where the information may have evolved or changed.

The 1th issue of "Write go Code problems" after the release has been a lot of gopher support and appreciation, this is my continued to write down the power! However, it is still to be emphasized that this series of articles reflects the author's knowledge of code writing and the evolution of code in practice. The code here may just be "intermediate", not an optimal outcome, but I'm just recording a thought process of the problem and the code. However, communication and criticism are very welcome.

I. Daily operations of DEP

Although DEP still has a high init failure rate in the country (because of some Qiang third-party package) snag, I, like the mainstream Gopher community and projects, hesitate to choose to use DEP in the code base. DEP has just released the 0.4.1 version this week, the biggest difference from previous versions is that DEP has released its official website and relatively complete documentation (to replace the humble, low-profile FAQ on the GitHub Project homepage), which is a sign that DEP continues to mature. However, it is still unknown when DEP can merge into the Go Tools chain. However, DEP continues to exist as a standalone tool for a long period of time until the merge is widely accepted in Go tools.

Package-dependent management tools do not require much presence in their daily development, and the features we need are powerful but interface "small", good for developers, less concerned about how they work, and DEP basically compliant. The three most important commands for DEP daily operations are DEP Init, dep ensure, and DEP status. In the article "First Glimpse of DEP," I've focused on the DEP init principle, which is not the point here, and we use an example of daily workflow that uses Dep.

1. DEP init empty Project

We can init an empty project or an initial frame of project, where Init is an empty project, as a follow-up example base:

➜  $GOPATH/src/depdemo $dep init -vGetting direct dependencies...Checked 1 directories for packages.Found 0 direct dependencies.Root project is "depdemo" 0 transitively valid internal packages 0 external packages imported from 0 projects(0)   ✓ select (root)  ✓ found solution with 0 packages from 0 projectsSolver wall times by segment:  select-root: 68.406µs        other:  9.806µs  TOTAL: 78.212µs➜  $GOPATH/src/depdemo $lsGopkg.lock    Gopkg.toml    vendor/➜  $GOPATH/src/depdemo $dep statusPROJECT  CONSTRAINT  VERSION  REVISION  LATEST  PKGS USED

DEP Init has three outputs: Gopkg.lock, GOPKG.TOML and vendor directories, where GOPKG.TOML (including example, but commented out) and vendor are empty, Gopkg.lock contains only some metadata for GPS use:

➜  $GOPATH/src/depdemo git:(a337d5b) $cat Gopkg.lock# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.[solve-meta]  analyzer-name = "dep"  analyzer-version = 1  inputs-digest = "ab4fef131ee828e96ba67d31a7d690bd5f2f42040c6766b1b12fe856f87e0ff7"  solver-name = "gps-cdcl"  solver-version = 1

2. General operation Cycle: for {Fill code, DEP ensure}

The next general step is to add code to project. Let's start by adding a Main.go file for the project, with the following source code:

// main.gopackage mainimport "fmt"func main() {    fmt.Println("depdemo")}

This code relies only on the FMT of the STD library and does not use third-party dependencies, so when we look at the current state through DEP status and use ensure to synchronize, we find that DEP has nothing to do:

➜  $GOPATH/src/depdemo $dep statusPROJECT  CONSTRAINT  VERSION  REVISION  LATEST  PKGS USED➜  $GOPATH/src/depdemo $dep ensure -vGopkg.lock was already in sync with imports and Gopkg.toml

All right. Let's add something "useful" to Main.go: the code that reads the TOML configuration file.

//data.tomlid = "12345678abcdefgh"name = "tonybai"city = "shenyang"// main.gopackage mainimport (    "fmt"    "log"    "github.com/BurntSushi/toml")type Person struct {    ID   string    Name string    City string}func main() {    p := Person{}    if _, err := toml.DecodeFile("./data.toml", &p); err != nil {        log.Fatal(err)    }    fmt.Println(p)}

After that, you can then perform the DEP status:

➜  $GOPATH/src/depdemo $dep statusLock inputs-digest mismatch due to the following packages missing from the lock:PROJECT                     MISSING PACKAGESgithub.com/BurntSushi/toml  [github.com/BurntSushi/toml]This happens when a new import is added. Run `dep ensure` to install the missing packages.input-digest mismatch

We see DEP status detecting "out of sync" in the project (the TOML package referenced in the code is not in Gopkg.lock) and we recommend that you use the DEP ensure command to do a sync.

Let's ensure (see ensure input and Output):

$GOPATH/src/depdemo git:(master) $dep ensure -vRoot project is "depdemo" 1 transitively valid internal packages 1 external packages imported from 1 projects(0)   ✓ select (root)(1)    ? attempt github.com/BurntSushi/toml with 1 pkgs; 7 versions to try(1)        try github.com/BurntSushi/toml@v0.3.0(1)    ✓ select github.com/BurntSushi/toml@v0.3.0 w/1 pkgs  ✓ found solution with 1 packages from 1 projectsSolver wall times by segment:     b-source-exists: 15.821158205s... ...  b-deduce-proj-root:       5.453µs  TOTAL: 16.176846089s(1/1) Wrote github.com/BurntSushi/toml@v0.3.0

Let's see what happens to the files in the project:

$git statusOn branch masterChanges not staged for commit:  (use "git add <file>..." to update what will be committed)  (use "git checkout -- <file>..." to discard changes in working directory)    modified:   Gopkg.lockUntracked files:  (use "git add <file>..." to include in what will be committed)    vendor/

You can see that the Gopkg.lock file and the vendor directory have changed:

$git diffdiff --git a/Gopkg.lock b/Gopkg.lockindex bef2d00..c5ae854 100644--- a/Gopkg.lock+++ b/Gopkg.lock@@ -1,9 +1,15 @@ # This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.+[[projects]]+  name = "github.com/BurntSushi/toml"+  packages = ["."]+  revision = "b26d9c308763d68093482582cea63d69be07a0f0"+  version = "v0.3.0"+ [solve-meta]   analyzer-name = "dep"   analyzer-version = 1-  inputs-digest = "ab4fef131ee828e96ba67d31a7d690bd5f2f42040c6766b1b12fe856f87e0ff7"+  inputs-digest = "25c744eb70aefb94032db749509fd34b2fb6e7c6041e8b8c405f7e97d10bdb8d"   solver-name = "gps-cdcl"   solver-version = 1$tree -L 2 vendorvendor└── github.com    └── BurntSushi

You can see that the TOML Package's dependency entry (version v0.3.0) has been added to the Gopkg.lock, input-digest the value of the metadata field has changed, and vendor the source of the TOML package, and the project has reached the "sync" state.

3. Adding constraints

In most cases, we're here to complete a cycle of DEP work flow, but if you need to add some constraints to the version of the third-party package, DEP Ensure-add will come in handy, For example: We want to use the v0.2.x version of the TOML package instead of the v0.3.0 version, and we need to add a constraint to the GITHUB.COM/BURNTSUSHI/TOML:

$dep ensure -v -add github.com/BurntSushi/toml@v0.2.0Fetching sources...(1/1) github.com/BurntSushi/toml@v0.2.0Root project is "depdemo" 1 transitively valid internal packages 1 external packages imported from 1 projects(0)   ✓ select (root)(1)    ? attempt github.com/BurntSushi/toml with 1 pkgs; at least 1 versions to try(1)        try github.com/BurntSushi/toml@v0.3.0(2)    ✗   github.com/BurntSushi/toml@v0.3.0 not allowed by constraint ^0.2.0:(2)        ^0.2.0 from (root)(1)        try github.com/BurntSushi/toml@v0.2.0(1)    ✓ select github.com/BurntSushi/toml@v0.2.0 w/1 pkgs  ✓ found solution with 1 packages from 1 projectsSolver wall times by segment:... ...  TOTAL: 599.252392ms(1/1) Wrote github.com/BurntSushi/toml@v0.2.0

Add constraint, a record is added to the GOPKG.TOML:

// Gopkg.toml[[constraint]]  name = "github.com/BurntSushi/toml"  version = "0.2.0"

The version of the TOML entry in Gopkg.lock is fallback to v0.2.0:

diff --git a/Gopkg.lock b/Gopkg.lockindex c5ae854..a557251 100644--- a/Gopkg.lock+++ b/Gopkg.lock@@ -4,12 +4,12 @@ [[projects]]   name = "github.com/BurntSushi/toml"   packages = ["."]-  revision = "b26d9c308763d68093482582cea63d69be07a0f0"-  version = "v0.3.0"+  revision = "bbd5bb678321a0d6e58f1099321dfa73391c1b6f"+  version = "v0.2.0" [solve-meta]   analyzer-name = "dep"   analyzer-version = 1-  inputs-digest = "25c744eb70aefb94032db749509fd34b2fb6e7c6041e8b8c405f7e97d10bdb8d"+  inputs-digest = "9fd144de0cc448be93418c927b5ce2a70e03ec7f260fa7e0867f970ff121c7d7"   solver-name = "gps-cdcl"   solver-version = 1$dep statusPROJECT                     CONSTRAINT  VERSION  REVISION  LATEST  PKGS USEDgithub.com/BurntSushi/toml  ^0.2.0      v0.2.0   bbd5bb6   v0.2.0  1

Vendor directory under the TOML package source code also back to v0.2.0 source. For the constituent syntax of constraint rules, you can refer to the DEP documentation.

4, Revendor/update Vendor

With the vendor mechanism, Revendor third-party dependent package versions or update vendor can be a recurring task because third-party dependency packages fix bugs or introduce features you need. For example: TOML package made some bugfix, and released the v0.2.1 version. In my depdemo, in order to fix these bugs together, I need to vendor the TOML package again. Before we added the constraint to the upgrade to the v0.2.1 version, so we don't need to reset the constraints, we only need to Revendor toml separately, you can use the DEP ensure-update command:

$dep ensure -v -update github.com/BurntSushi/tomlRoot project is "depdemo" 1 transitively valid internal packages 1 external packages imported from 1 projects(0)   ✓ select (root)(1)    ? attempt github.com/BurntSushi/toml with 1 pkgs; 7 versions to try(1)        try github.com/BurntSushi/toml@v0.3.0(2)    ✗   github.com/BurntSushi/toml@v0.3.0 not allowed by constraint ^0.2.0:(2)        ^0.2.0 from (root)(1)        try github.com/BurntSushi/toml@v0.2.0(1)    ✓ select github.com/BurntSushi/toml@v0.2.0 w/1 pkgs  ✓ found solution with 1 packages from 1 projectsSolver wall times by segment:  b-list-versions: 1m18.267880815s  .... ...  TOTAL: 1m57.118656393s

Because the real toml does not have a v0.2.1 version and there is no v0.2.x version, our DEP ensure-update does not actually get the data. Vendor and Gopkg.lock have not changed.

5, DEP daily Operation summary

The following diagram, which contains the three DEP daily operations, visually shows the changes to the project after different operations:

"工欲善其事, its prerequisite", skilled in the daily operation of DEP to improve development efficiency.

Ii. an implementation of the "Timeout pending exit" framework

Most of the time, we start in the program with multiple goroutine that collaborate to complete the application's business logic, such as:

func main() {    go producer.Start()    go consumer.Start()    go watcher.Start()    ... ...}

Start easy stop hard! When the program is going to exit, the most brutal way is to quit, regardless of 3,721, main goroutine directly; elegant way, also *nix system is usually the practice is: inform the Goroutine to exit, and then wait for a period of time before the real exit. A rude direct exit may result in corruption, incomplete, or loss of business data. While waiting for a timeout cannot completely avoid "loss", it gives each goroutine a "save data" opportunity to reduce the amount of damage possible.

But these goroutine patterns are likely to be different, some are server, some may be client workers or their manager, so it seems difficult to fully manage their start, run, and exit with a unified framework, so we narrow the "interface" and we only do "timeout waiting to exit" ”。 We define a interface:

type GracefullyShutdowner interface {    Shutdown(waitTimeout time.Duration) error}

In this way, any type that implements the interface can be notified of exit when the program exits and has the opportunity to do the final cleanup before exiting. A similar HTTP is provided here. Handlerfunc type shutdownerfunc for converting ordinary function into an instance of a type that implements Gracefullyshutdowner interface:

type ShutdownerFunc func(time.Duration) errorfunc (f ShutdownerFunc) Shutdown(waitTimeout time.Duration) error {    return f(waitTimeout)}

1. Concurrent exit

There are at least two types of exits, one is the concurrent exit, the exit order of each goroutine in the exit sequence has no effect on the data processing, and the other is sequential exit, that is, the exit between each goroutine must be in a certain order. Let's start with a concurrent exit. On the code!

// shutdown.gofunc ConcurrencyShutdown(waitTimeout time.Duration, shutdowners ...GracefullyShutdowner) error {    c := make(chan struct{})    go func() {        var wg sync.WaitGroup        for _, g := range shutdowners {            wg.Add(1)            go func(shutdowner GracefullyShutdowner) {                shutdowner.Shutdown(waitTimeout)                wg.Done()            }(g)        }        wg.Wait()        c <- struct{}{}    }()    select {    case <-c:        return nil    case <-time.After(waitTimeout):        return errors.New("wait timeout")    }}

We pass the implementation of each Gracefullyshutdowner interface into the Concurrencyshutdown function as a variable-length parameter. The Concurrencyshutdown function is also simple to implement by:

• Start a goroutine for each shutdowner to implement concurrent exits, and pass the timeout parameter into the shutdown method of Shutdowner;
• Sync. Waitgroup waits for each goroutine exit in the outer layer;
• The channel and time are indicated by the Select one exit. After the timer channel is returned to determine whether the exit is normal or timed out.

The specific use of this function can be referenced by: Shutdown_test.go.

//shutdown_test.gofunc shutdownMaker(processTm int) func(time.Duration) error {    return func(time.Duration) error {        time.Sleep(time.Second * time.Duration(processTm))        return nil    }}func TestConcurrencyShutdown(t *testing.T) {    f1 := shutdownMaker(2)    f2 := shutdownMaker(6)    err := ConcurrencyShutdown(time.Duration(10)*time.Second, ShutdownerFunc(f1), ShutdownerFunc(f2))    if err != nil {        t.Errorf("want nil, actual: %s", err)        return    }    err = ConcurrencyShutdown(time.Duration(4)*time.Second, ShutdownerFunc(f1), ShutdownerFunc(f2))    if err == nil {        t.Error("want timeout, actual nil")        return    }}

2. Serial exit

With the concurrent exit as the basis, serial exit is also very simple!

//shutdown.gofunc SequentialShutdown(waitTimeout time.Duration, shutdowners ...GracefullyShutdowner) error {    start := time.Now()    var left time.Duration    for _, g := range shutdowners {        elapsed := time.Since(start)        left = waitTimeout - elapsed        c := make(chan struct{})        go func(shutdowner GracefullyShutdowner) {            shutdowner.Shutdown(left)            c <- struct{}{}        }(g)        select {        case <-c:            //continue        case <-time.After(left):            return errors.New("wait timeout")        }    }    return nil}

One problem with serial exit is the determination of waittimeout, because this timeout is the sum of all goroutine exit times. In the above code, I pass each lefttime to the next Goroutine shutdown method to execute, and the external select also uses this left as the value of timeout. It's easier to control concurrencyshutdown,sequentialshutdown than to say it in detail here.

3. Summary

This is a usable, out-of-the-way implementation, but there is a lot of room for improvement, such as: Consider getting each shutdowner. The return value after shutdown (error) is left to everyone's own consideration.

TestCase's setup and teardown

The go language comes with a testing framework, which turns out to be one of the great advantages of the go language, and Gopher is also very fond of this testing bag. But testing this is complicated, and some scenarios require our own brains to implement the required functionality in the standard testing framework, such as when the test code needs to access an external database, Redis, or connect to the remote server. In this case, many people think of the mock, yes. Mock technology can solve these problems to some extent, but if you use mock technology, the business code has to be a layer of abstraction for test, which improves the difficulty of understanding the code, and sometimes it's not as straightforward to access the real external environment.

The pros and cons of these two approaches are not discussed here, but we'll just discuss how we can test if we're accessing the real environment in testing. In the Classic unit test framework, we often see the setup and teardown two methods, which are used to initialize the execution environment of TestCase before TestCase executes and to clean the execution environment after testcase execution. To ensure that each of the two testcase between the independent, non-interference. In a real-world environment, we can also use Setup and teardown to initialize and clean up the real-world case dependencies for each testcase.

Setup and teardown also have levels, with global, testsuite, and testcase levels. In go, under the standard testing framework, we are exposed to both global and testcase levels. Support for global-level setup and teardown in Go goes back to go 1.4,go 1.4 introduces the Testmain method, which enables you to add a custom Setup to your test code before many testcase execute. and perform teardown operations after testing execution, for example:

func TestMain(m *testing.M) {    err := setup()    if err != nil {        fmt.Println(err)        os.Exit(-1)    }    r := m.Run()    teardown()    os.Exit(r)}

But at the testcase level, the Go testing package does not provide method support. At the Gophercon conference in 2017, Hashicorp's founder, Mitchell Hashimoto, made a keynote speech titled "Advanced Testing in Go", A more elegant approach to setup and Teawdown for TestCase is presented in this document:

//setup-teardown-demo/foo_test.gopackage foo_testimport (    "fmt"    "testing")func setUp(t *testing.T, args ...interface{}) func() {    fmt.Println("testcase setUp")    // use t and args    return func() {        // use t        // use args        fmt.Println("testcase tearDown")    }}func TestXXX(t *testing.T) {    defer setUp(t)()    fmt.Println("invoke testXXX")}

This scheme takes full advantage of the function of the first-class type and the function of closures, each testcase can customize its own setup and teardown, or use common setup and teardown to perform the following effects:

$go test -v .=== RUN   TestXXXtestcase setUpinvoke testXXXtestcase tearDown--- PASS: TestXXX (0.00s)PASSok      github.com/bigwhite/experiments/writing-go-code-issues/2nd-issue/setup-teardown-demo    0.010s

Iv. Error Handling

Originally wanted to code some of the words about Go error handling, but found yourself in 2015 wrote an old article "Go Language Error Processing", on the Go error treatment of all aspects of the summary is very comprehensive. Even today, this is certainly due to the existence of the GO1 compatibility specification. So for those of you who are interested in this, please take a go to the article on the language error handling.

Note: Please download the sample code for this article.

Weibo: @tonybai_cn
Public Number: Iamtonybai
Github.com:https://github.com/bigwhite

Appreciated:

2018, Bigwhite. All rights reserved.