Preliminary study on Golang modules

Today the sky just lit up, get up to see Golang 1.11 official release, with two important features: modules and webassembly.

This blog as long as the Modules,congjava turn Golang classmate is certainly the Golang package management is full of frustration, I have also introduced glide in the blog, and introduced DEP, and now we upgrade the introduction modules.

What is modules

Now it's all about modules, so what is it?
Take a look at the document Modules, module versions, and more:

A module is a collection of related Go packages. Modules are the unit of source code interchange and versioning. The go command has direct support for working with modules, including recording and resolving dependencies on other modules. Modules replace the old GOPATH-based approach to specifying which source files are used in a given build.

Translation:

模块是相关Go包的集合。modules是源代码交换和版本控制的单元。 go命令直接支持使用modules，包括记录和解析对其他模块的依赖性。modules替换旧的基于GOPATH的方法来指定在给定构建中使用哪些源文件。

You can get two important information:

• Go command line supports modules operation
• Modules used to replace Gopath.

We do not need to worry too much, Golang 1.11 version only refers to the preliminary support of modules, before the old Gopath still can continue to use, some say it is in the Golang 1.12 removal, but I feel a little early, after all, people's inertia is not so easy to change.

How to use Modules

Modules is a new feature, then you need to support the new Golang version, can be downloaded to the official website, it must be go 1.11 and above (when you write the blog, go 1.11 just came out). So the deployment is here to say, I believe that beginners also know how to do.

Does anyone remember the environment variables that vendor just came out with Golang GO15VENDOREXPERIMENT ? Now modules out, as is customary to provide an environment variable GO111MODULE , the three 1 of this variable is too magical.

Go111module

GO111MODULECan be set to one of three string values: Off,on or Auto (default).

• Off, the GO command is not supported with the new module. It looks for vendor directories and Gopath to find dependencies; that is, continue to use "Gopath mode".
• On, the GO command needs to use the module, and go ignores the Gopath and vendor folders, depending only on the go.mod download.
• Auto or not set, the GO command enables or disables module support based on the current directory. Module support is enabled only if the current directory is outside of GOPATH/SRC and itself contains a Go.mod file or is located in a directory that contains go.mod files.

Defining a Module

Who doesn't know how to use it when it starts? But go has provided me with a tool that can be entered in the console:

go help modules

See a bunch of document output, look at all the headache, a moment we briefly explain the focus, now the first operation.

QIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ export GO111MODULE= On #开启modulesqiang @DESKTOP -2a835p9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ go mod Init Gitlab.luojilab.com/zeroteam/ddkafka # Create go.modgo:creating new Go.mod:module gitlab.luojilab.com/zeroteam/ DDKAFKAQIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ ls # really created, Google Dafa good readme.md go.mod models.go mq_interface.go Sarama segmentioqiang@desktop-2a835p9 mingw64/d/code/gopath/src /gitlab.luojilab.com/zeroteam/ddkafka (module) $ cat Go.mod # Look what's inside. Module gitlab.luojilab.com/zeroteam/ DDKAFKAQIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ cd Segmentio /QIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module) $ Go test # Take a look at go:finding github.com/segmentio/kafka-go latestgo:finding GitHub.Com/golang/glog latestgo:downloading Github.com/golang/glog v0.0.0-20160126235308-23def4e6c14bgo:downloading Github.com/segmentio/kafka-go V0.0.0-20180716203113-48C37F796910QIANG@DESKTOP-2A835P9 mingw64/d/code/gopath/src/ Gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module) $ go List-mgitlab.luojilab.com/zeroteam/ddkafka

Careful classmate must be able to find that execution go mod init [module] uses go.mod only one line of information module gitlab.luojilab.com/zeroteam/ddkafka , when the go build, go test, go List command will automatically generate the Require statement according to the dependent dependency.

Now, how to define a modules,modules is defined by the Go source directory structure, and if the directory contains go.mod files, this directory is called the module root directory. The module root and its subdirectories all go packages belong to the modules, but if the subdirectory contains its own go.mod file, it belongs to the modules. To give an example:

qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module)$ tree.|-- README.md|-- go.mod|-- go.sum|-- models.go|-- mq_interface.go|-- sarama|   |-- sarama_consumer.go|   |-- sarama_consumer_test.go|   |-- sarama_producer.go|   `-- sarama_producter_test.go`-- segmentio    |-- segmention_Consumer.go    |-- segmention_consumer_test.go    |-- segmention_producer.go    `-- segmention_producter_test.go

gitlab.luojilab.com/zeroteam/ddkafkaThe directory contains the Go.mod file, so its subdirectories sarama and segmentio all belong to the gitlab.luojilab.com/zeroteam/ddkafka module, but if segmentio Go.mod is added to the directory, then Segmentio is no longer affiliated with the gitlab.luojilab.com/zeroteam/ddkafka module.

So dependencies are downloaded to where, you can open the directory you $GPATH/pkg/mod can see.

Master module and build list

The main module and the build list are temporarily translated into the main modules and build lists. The "main module" is the module that contains the directory where the Go command is run. The GO command finds the Go.mod in the current directory, or the parent directory of the current directory, or the grandfather directory, in turn recursively.

Go.mod files can be used by require,replace and exclude statements for the exact package set.

• The Require statement specifies the dependency module
• The Replace statement can replace the dependency module
• The exclude statement can ignore the dependency module

go list, you can view the current dependencies and versions.

qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module)$ ls  # 这是模块的子目录segmention_Consumer.go  segmention_consumer_test.go  segmention_producer.go  segmention_producter_test.goqiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module)$ go list -m #主模块的打印路径gitlab.luojilab.com/zeroteam/ddkafkaqiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module)$ go list -m -f={{.Dir}} #print主模块的根目录D:\code\gopath\src\gitlab.luojilab.com\zeroteam\ddkafkaqiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/segmentio (module)$ go list -m all # 查看当前的依赖和版本信息gitlab.luojilab.com/zeroteam/ddkafkagithub.com/golang/glog v0.0.0-20160126235308-23def4e6c14bgithub.com/segmentio/kafka-go v0.0.0-20180716203113-48c37f796910

Go MoD command

go modThe command can be used before go mod init , we will list the commonly used go mod commands:

• Go mod init: Initialize modules
• Go mod download: Download modules to local cache
• Go mod edit: edited go.mod file with options-json,-require and-exclude, you can use the Help Go Helper mod edit
• Go MoD graph: Print module requirements diagram in text mode
• Go mod tidy: Remove errors or unused modules
• Go mod vendor: Generate Vendor Directory
• Go MoD verify: Verify that dependencies are correct
• Go mod why: Find dependencies

QIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ go mod edit-json{                        "Module": {"Path": "Gitlab.luojilab.com/zeroteam/ddkafka"}, "Require": [{ "Path": "Github.com/golang/glog", "Version": "v0.0.0-20160126235308-23def4e6c                        14b "}, {" Path ":" Github.com/segmentio/kafka-go ", "Version": "v0.0.0-20180716203113-48c37f796910"}], "Exclude": null, "Replace": NULL}QIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ cat Go.modmodule Gitlab.luojilab.com/zeroteam/ddkafkarequire (Github.com/golang/glog v0.0.0-20160126235308-23def4e6c1 4b github.com/segmentio/kafka-go v0.0.0-20180716203113-48c37f796910) qiang@desktop-2a835p9 mingw64/d/code/gopath/s Rc/gitlab.luojilab.com/zeroteam/ddkafka (MoDule) $ go mod edit-require=github.com/shopify/sarama@masterqiang@desktop-2a835p9 mingw64/d/code/gopath/src/ Gitlab.luojilab.com/zeroteam/ddkafka (module) $ cat go.modmodule Gitlab.luojilab.com/zeroteam/ddkafkarequire (Githu B.com/shopify/sarama Master Github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b github.com/segmentio/k Afka-go v0.0.0-20180716203113-48c37f796910) Qiang@desktop-2a835p9 mingw64/d/code/gopath/src/gitlab.luojilab.com/ Zeroteam/ddkafka (module) $ Go mod Vendor # boot verdongo:downloading Github.com/shopify/sarama V1.17.1-0.20180820172058-647feef69a1ago:finding Github.com/davecgh/go-spew/spew latestgo:finding github.com/ Eapache/queue v1.1.0go:finding github.com/eapache/go-xerial-snappy latestgo:finding github.com/eapache/ Go-resiliency/breaker latestgo:finding github.com/rcrowley/go-metrics latestgo:downloading github.com/rcrowley/ Go-metrics v0.0.0-20180503174638-e2704e165165go:finding Github.com/bsm/sarama-cluster V2.1.15+incompatiblego:doWnloading github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21go:downloading github.com/bsm/ Sarama-cluster v2.1.15+incompatiblego:downloading github.com/eapache/queue v1.1.0go:finding github.com/eapache/ Go-resiliency v1.1.0go:downloading github.com/eapache/go-resiliency v1.1.0go:finding Github.com/davecgh/go-spew V1.1.1go:downloading Github.com/davecgh/go-spew v1.1.1go:finding github.com/pierrec/lz4 v2.0.3+incompatiblego: Downloading github.com/pierrec/lz4 v2.0.3+incompatiblego:finding github.com/golang/snappy latestgo:downloading Github.com/golang/snappy V0.0.0-20180518054509-2E65F85255DBQIANG@DESKTOP-2A835P9 mingw64/d/code/gopath/src/  Gitlab.luojilab.com/zeroteam/ddkafka (module) $ lsreadme.md go.mod go.sum models.go mq_interface.go Sarama segmentio VENDORQIANG@DESKTOP-2A835P9 Mingw64/d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka (module) $ go mod Verifyall Modules Verifiedqiang@desktop-2a835p9 mingw64/d/code/gopath/src/gitlab.luojilab.com/zEroteam/ddkafka (module) $ go mod whygo:finding github.com/onsi/ginkgo/extensions/table latestgo:finding github.com/ Onsi/ginkgo v1.6.0go:finding github.com/shopify/toxiproxy/client latestgo:finding github.com/onsi/gomega v1.4.1go: Downloading Github.com/onsi/gomega v1.4.1go:downloading github.com/onsi/ginkgo v1.6.0go:finding github.com/onsi/ Ginkgo/extensions latestgo:finding github.com/shopify/toxiproxy v2.1.3+incompatiblego:downloading github.com/ Shopify/toxiproxy v2.1.3+incompatiblego:finding github.com/hpcloud/tail v1.0.0go:finding github.com/golang/ Protobuf/proto latestgo:finding gopkg.in/yaml.v2 v2.2.1go:downloading github.com/hpcloud/tail v1.0.0go:downloading Gopkg.in/yaml.v2 v2.2.1go:finding github.com/golang/protobuf v1.2.0go:downloading github.com/golang/protobuf V1.2.0go:finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405go:downloading gopkg.in/check.v1 V0.0.0-20161208181325-20d25e280405go:finding gopkg.in/tomb.v1 latestgo:finding gopkg.in/fsnotify.V1 v1.4.7go:downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7go:downloading gopkg.in/fsnotify.v1 V1.4.7go:finding github.com/fsnotify/fsnotify v1.4.7go:downloading github.com/fsnotify/fsnotify v1.4.7# Gitlab.luojilab.com/zeroteam/ddkafkagitlab.luojilab.com/zeroteam/ddkafka

Go mod and get

Go get this command everyone should not be unfamiliar, this is the download go dependency package based on, download go 1.11 out, go get command also with the Times, support modules. Go get to update module:

• Running Go Get-u will be upgraded to the latest minor or revised version
• Running Go Get-u=patch will be upgraded to the latest revision (for example, it will be upgraded to version 1.0.1, but will not be upgraded to 1.1.0 version)
• Running go get package@version will be upgraded to the specified version number

Run go get if there is a version change, then the Go.mod file will also change.

How to handle being wall

I originally thought that go 1.11 version modules does not support the proxy function, but has been verified and is supported. The command used is the one previously mentioned replace . There is a passage in the document:

The -replace=old[@v]=new[@v] and -dropreplace=old[@v] flagsadd and drop a replacement of the given module path and version pair.If the @v in old@v is omitted, the replacement applies to all versionswith the old module path. If the @v in new@v is omitted, the new pathshould be a local module root directory, not a module path.Note that -replace overrides any existing replacements for old[@v].

Probably translated:

-replace=old[@v]=new[@v]和-dropreplace=old[@v]标志添加和删除给定模块路径和版本对的替换。如果省略了old@v中的@v，则替换适用于所有使用旧模块路径的版本。如果new@v中的@v被省略，新路径应该是本地模块根目录，而不是模块路径。注意-replace会覆盖任何现有的旧替换[@v]。

Here's a demonstration: Create a new main file with the following code:

package mainimport ("errors""fmt""golang.org/x/net/netutil""net""time")var errFake = errors.New("fake error from errorListener")type errorListener struct {net.Listener}func main() {donec := make(chan bool, 1)go func() {const n = 2ll := netutil.LimitListener(errorListener{}, n)for i := 0; i < n+1; i++ {_, err := ll.Accept()if err != errFake {fmt.Errorf("Accept error = %v; want errFake", err)}}donec <- true}()select {case <-donec:case <-time.After(5 * time.Second):fmt.Errorf("timeout. deadlock?")}}

You can see that we have introduced golang.org/x/net/netuti , unfortunately, that this is not accessible to the class library.

qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ go build  # 试试是否可以下载build gitlab.luojilab.com/zeroteam/ddkafka/cmd: cannot find module for path golang.org/x/net/netutilqiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ go mod edit -require=golang.org/x/net@v1.2.3 # modules中加上依赖，版本是乱给的qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ go build  # 还是不行go: golang.org/x/net@v1.2.3: unrecognized import path "golang.org/x/net" (https fetch: Get https://golang.org/x/net?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)go: error loading module requirements

Ended in failure! But we still have a replace way to download it first.

qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/github.com/golang$ git clone https://github.com/golang/net.gitCloning into 'net'...remote: Counting objects: 7876, done.remote: Compressing objects: 100% (55/55), done.remote: Total 7876 (delta 37), reused 42 (delta 19), pack-reused 7802RReceiving objects: 100% (7876/7876), 6.35 MiB | 812.00 KiB/s, done.Resolving deltas: 100% (5443/5443), done.qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/github.com/golang/net (master)$ go mod init github.com/golang/net  #创建modulesgo: creating new go.mod: module github.com/golang/netqiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/github.com/golang/net (master)$ cat go.modmodule github.com/golang/net

Back to the original project:

qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ go mod edit -require=golang.org/x/net@v1.2.3 # qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$  go mod edit -replace=golang.org/x/net@v1.2.3=/d/code/gopath/src/github.com/golang/net # 使用replace替换到指定的目录了qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ go build  # 这一次成功了qiang@DESKTOP-2A835P9 MINGW64 /d/code/gopath/src/gitlab.luojilab.com/zeroteam/ddkafka/cmd (module)$ lscmd.exe  main.go

At last

Finally, the latest features are not recommended for immediate use to the line, it is best to wait, and so on after the iteration of one or two versions, after the best practice comes out, after all, now support modules mode of the class library is really not much.

Appendix

• https://tip.golang.org/cmd/go/#hdr-modules__module_versions__and_more
• Https://roberto.selbach.ca/intro-to-go-modules