First Glimpse DEP

This is a creation in Article, where the information may have evolved or changed.

The basic unit for the Go Language program is organization and construction, but the go language official does not offer a "decent" package Management tool (packet management tools). With the growing use of the go language worldwide, the lack of official package management tools has become increasingly prominent.

After the Gophercon conference in 2016, a commitee, designed to improve go package management, was set up under the go official organization to address the various issues that go encountered with the pack management. After a variety of brain cavities and discussions, the Commitee released a "package Management proposal" several months later and launched the design and development of DEP, which is most likely to be accepted as the Official Pack management tool. The DEP project was officially opened in early 2017. As of now, DEP has released the v0.1.0 version and is in the Alpha beta phase.

It can be said that the progress of DEP is pretty fast. DEP's current manifest and lock file format has been stable and guaranteed backwards compatibility, according to DEP official. At the same time, DEP implements Bootstrap, which means that DEP uses itself as its own package management tool. Because of the "special identity" of DEP, although DEP is still far from maturity, the progress of DEP attracts many gopher, and many organizations have begun to migrate the package management tool to DEP for early testing of DEP.

Here, I also intend to "taste fresh", in this article and everyone to spy and try out dep.

First, the evolution of Go package management history

1. Go get

Before looking at DEP, let's take a quick look at the evolution of the Go Language pack management. The first is go get.

It is really convenient for the go language novice to feel the go language package when first touching the go language: Just a large number of Go package on the go get xxx,github.com can be used with you. But with the depth of use of Go language, people will find that go get to bring us convenience, but also bring a lot of trouble. Go get is essentially a high-level wrapper of these VCs tools such as Git, Hg, and so on. For the go package that uses git, the essence of Go get is to clone packages git to a specific local directory ($GOPATH/src), while go get automatically resolves the dependency of the packet and automatically downloads the dependent package.

The go get mechanism is designed to be largely rooted in the development model of a single root code warehouse within Google, and it seems that the code on the master branch of each project/repository within Google is considered stable, so go Get only supports obtaining the latest code on Master branch, without the ability to specify version, branch, or revision. In a world outside Google, such practices can be inconvenient for Gopher: dependent third-party packages are always changing. Once a third-party package submits code that is not properly build or incompatible with the interface, the relying party is immediately affected.

And the gopher just want their projects to rely on the third party package can be controlled by their own, rather than random changes. In this way, GODEP, GB, glide and other third-party package management tools appear.

Take the most widely used GODEP as an example. In order for a third party to rely on the package to "stabilize", implement the project's reproduceble BUILD,GODEP The version information of the project's current dependency package is recorded in Godeps/godeps.json, and the dependent version of the package is stored in the Godeps/_ The workspace. At compile time (GODEP go build) GODEP lets the go compiler use a specific version of a third-party package that caches the dependencies of the project under Godeps/_workspace by temporarily modifying the GOPATH environment variable. This ensures that the project is no longer subject to changes in the latest code on the master branch of the dependent third-party package.

However, GODEP's "versioning" is essentially done by caching a snapshot of a third-party library's revision, which is still difficult to manage. At the same time, through the Gopath "rescue" way to achieve the use of the Godeps/_workspace in the third-party library snapshot to compile and not compatible with the Go native compiler, you must use GODEP go xxx.

To do this, go further introduces the vendor mechanism to reduce the mental burden of gopher on package management issues .

2. Vendor mechanism

Go Team has also been concerned about the issue of the Go Language pack dependency, especially in the go 1.5 implementation of the case, the official also in the 1.5 version of the introduction of the vendor mechanism. The vendor mechanism is Russ Cox's emergency addition to go in a experiment feature status in the Go 1.5 release (go 1.6 out of experiment identity). Vendor standardizes the storage location of the third-party libraries that the project relies on (no longer need to godeps/_workspace), and does not have to "rescue" the Gopath environment variables, and go compiler native first senses and uses a third-party package that is cached under vendor.

But even with vendor support, the management of third-party-dependent packages in vendor is still not standardized, either manually or with third-party package management tools such as GODEP. Now the Go code itself has introduced vendor, but the go project itself manages the code in vendor manually, and the go itself does not use any third-party package management tools.

Digression: As a standard library of a language, it should be the root dependency of all lib dependencies used by developers using this language. But in go, the Go standard library actually also relies on the golang.org/x/directory of the package, since it can be Std lib dependent, then it is mature, then why not the X stable library into the Std lib? That's a bit confusing.

~/.bin/go18/src/vendor/golang_org/x]$lscrypto/    net/    text/

From the go official point of view, the next step in the official go package-dependent solution should be to solve the problem of how third-party packages under vendor are managed: relying on packet analysis, logging, and acquisition, to achieve the reproducible build of the project. DEP is used to do this.

Ii. introduction of DEP

The lead figure for Go package Management Commitee is Peter Bourgon, the Go-kit author of the MicroServices framework, but the current leading DEP development is the author of Sam Boyer,sam and the DEP underlying package dependency analysis engine-gps.

Unlike some of the other third-party go package management tools, DEP has been commitee thoughtful before active dev, including: Features, user story, and so on have been pre-designed. If you read these documents, you may find it quite complicated to solve the problem of package dependencies. However, for the users of these tools, we are faced with some very simplified interface.

1. Installation DEP

DEP is a standard go CLI program that executes a command to complete the installation:

# go get -u github.com/golang/dep/cmd/dep# dep helpdep is a tool for managing dependencies for Go projectsUsage: dep <command>Commands:  init    Initialize a new project with manifest and lock files  status  Report the status of the project's dependencies  ensure  Ensure a dependency is safely vendored in the project  prune   Prune the vendor tree of unused packagesExamples:  dep init                          set up a new project  dep ensure                        install the project's dependencies  dep ensure -update                update the locked versions of all dependencies  dep ensure github.com/pkg/errors  add a dependency to the projectUse "dep help [command]" for more information about a command.

In my test environment, the go version of 1.8;DEP is the commit d31c621c3381b9bebc7c10b1ac7849a96c21f2c3.

Note: Because DEP is also in the active dev process and is in the alpha test phase, the DEP commands, command behavior, and output results that are performed in this article are likely to change, or even vary, in subsequent DEP releases.

2. DEP General workflow

Once you've installed DEP, let's look at a general workflow that uses DEP. We first prepare a demo program:

//depdemo/main.gopackage mainimport (    "net/http"    "go.uber.org/zap"    "github.com/beego/mux")func main() {    logger, _ := zap.NewProduction()    defer logger.Sync()    sugar := logger.Sugar()    mx := mux.New()    mx.Handler("GET", "/", http.FileServer(http.Dir(".")))    sugar.Fatal(http.ListenAndServe("127.0.0.1:8080", mx))}

A) DEP init

If a project is to use DEP for package management, you first need to perform DEP init under the root of this project. Here, we are depdemo to the DEP.

Under the Depdemo directory, perform DEP init:

# DEP init-vsearching Gopath for projects ... Using Master as constraint for direct DEP Github.com/beego/mux Locking in master (626AF65) for direct DEP Github.com/beeg O/muxfollowing dependencies were not found in Gopath.  DEP would use the most recent versions of these projects. Go.uber.org/zaproot project is "Github.com/bigwhite/experiments/depdemo" 1 transitively valid internal packages 2 External packages imported from 2 projects (0) ✓select (root) (1)? Attempt Github.com/beego/mux with 1 pkgs; At least 1 versions to try (1) Try Github.com/beego/mux@master (1) ✓select github.com/beego/mux@master w/1 pkgs (2 )    ? Attempt go.uber.org/zap with 1 pkgs; Versions to try (2) Try go.uber.org/zap@v1.4.0 (2) ✓select go.uber.org/zap@v1.4.0 W/7 pkgs (3)? Attempt go.uber.org/atomic with 1 pkgs; 6 versions to try (3) Try go.uber.org/atomic@v1.2.0 (3) ✓select go.uber.org/atomic@v1.2.0 w/1 pkgs✓found solut Ion with 9 packages from 3 projectssolver wall times by segment:b-source-exists:1.090607387s b-deduce-proj-root:288.126482ms b-list-pkgs:131.059753ms B-gmal:114.716587ms select-atom:337.787µs satisfy:298.743µs select-root:292 .889µs new-atom:257.256µs B-list-versions:42.408µs other:22.307µs total:1.625 761599s

At the current stage, the DEP init command is not performing very efficiently, so you need to wait patiently for a while. If your project relies on a lot of external packages, the wait can be a long time. And because DEP downloads dependent packages, for domestic friends, if you download a package outside of Qiang, Dep may "block" there!

DEP Init roughly does a few things:

• Using GPS to analyze packet dependencies in the current code package;
• The direct dependency of the analyzed project package (that is, the Main.go explicit import third-party package, the direct dependency) constraint (constraint) is written to the gopkg.toml file in the project root directory;
• All third-party packages that are dependent on the project (including direct dependency and transitive dependency transitive dependency) are written to the Gopkg.lock file in order to meet the latest version/branch/revision information within the constraints of GOPKG.TOML;
• Create the root vendor directory and, with Gopkg.lock as input, download the package (accurate checkout to revision) below the project root vendor.

After you finish DEP init, DEP generates several files in the current directory:

├── Gopkg.lock├── Gopkg.toml├── main.go└── vendor/

Let's take a look:

GOPKG.TOML:

[[constraint]]  branch = "master"  name = "github.com/beego/mux"[[constraint]]  name = "go.uber.org/zap"  version = "1.4.0"

GOPKG.TOML recorded the two direct Dependency:mux and Zap of Depdemo/main.go. With the analysis of the GPS (see the detailed analysis process log for the output at init execution above), DEP determines the dependent version constraint: The master branch of the MUX, the 1.4.0 version of Zap.

The resulting Gopkg.lock records the latest available versions of all the dependencies depdemo/main.go under the above constraints:

Gopkg.lock:[[projects]]  branch = "master"  name = "github.com/beego/mux"  packages = ["."]  revision = "626af652714cc0092f492644e298e5f3ac7db31a"[[projects]]  name = "go.uber.org/atomic"  packages = ["."]  revision = "4e336646b2ef9fc6e47be8e21594178f98e5ebcf"  version = "v1.2.0"[[projects]]  name = "go.uber.org/zap"  packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"  version = "v1.4.0"[solve-meta]  analyzer-name = "dep"  analyzer-version = 1  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"  solver-name = "gps-cdcl"  solver-version = 1

The vendor directory is the local clone of each dependent package in the lock file:

# tree -L 2 vendorvendor├── github.com│   └── beego└── go.uber.org    ├── atomic    └── zap

Now that DEP Init is complete and the dependent packages have been vendor, you can use go build/install to build the program.

b), submit gopkg.toml and Gopkg.lock

If you have no objection to the various constraints and dependent versions that DEP automatically parses, you can submit gopkg.toml and Gopkg.lock as part of the project's source code to the codebase. So that when someone downloads your code, you can download the third-party package version of the lock file directly from DEP, and there is vendor. This makes the dependent library of the project build consistent in theory and implements the Reproduceable build.

Do I need to submit the dependency package code under vendor to the code warehouse? It depends on you. The benefit of submitting vendor is that even without DEP, you can implement a true reproduceable build. But vendor commits can make your codebase huge, and when you update vendor, a lot of diff will affect your code review. The following content we do not submit vendor as a precondition.

c), DEP ensure

Now our Depdemo have joined GOPKG.TOML and Gopkg.lock. At this point, if you Depdemo clone to your local, you still cannot do reproduceable build, because vendor does not exist yet. At this point we need to execute the following command to build the vendor directory and synchronize the packages from the data in GOPKG.TOML and Gopkg.lock:

# dep ensure# ls -FGopkg.lock  Gopkg.toml  main.go  vendor/

Once the ensure is successful, you can make a reproduceable build.

We can view current dependencies (including direct and transitive dependency) through DEP status:

# dep statusPROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USEDgithub.com/beego/mux  branch master  branch master  626af65   626af65  1go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7

d) Specify constraints

Is the constraint in the gopkg.toml generated by DEP init what we expected? That's not necessarily true. For example: We will manually change the binding of Zap to 1.3.0:

//Gopkg.toml... ...[[constraint]]  name = "go.uber.org/zap"  version = "<=1.3.0"

After you perform DEP ensure, review the status:

# dep statusPROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USEDgithub.com/beego/mux  branch master  branch master  626af65   626af65  1go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1go.uber.org/zap       <=1.3.0         v1.4.0         fab4530   fab4530  7

However, the ZAP version in Gopkg.lock is still v1.4.0 and has not been modified. To update the data under lock and vendor, we need to add a-update parameter to ensure:

# dep ensure -update# git diff Gopkg.lockdiff --git a/depdemo/Gopkg.lock b/depdemo/Gopkg.lockindex fce53dc..7fe3640 100644--- a/depdemo/Gopkg.lock+++ b/depdemo/Gopkg.lock@@ -16,12 +16,12 @@ [[projects]]   name = "go.uber.org/zap"   packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]-  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"-  version = "v1.4.0"+  revision = "6a4e056f2cc954cfec3581729e758909604b3f76"+  version = "v1.3.0" [solve-meta]   analyzer-name = "dep"   analyzer-version = 1-  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"+  inputs-digest = "b09c1497771f6fe7cdfcf61ab1a026ccc909f4801c08f2c25f186f93f14526b0"   solver-name = "gps-cdcl"   solver-version = 1

-update let DEP Ensure attempts to secure and synchronize the data in the Gopkg.lock and vendor directories, changing the version of Zap under Gopkg.lock to the maximum of the GOPKG.TOML constraint, which is v1.3.0, while updating the Zap code under vendor.

e) Specify dependent

We can also update dependency directly, which will affect the data under Gopkg.lock and vendor, but GOPKG.TOML will not be modified:

# dep ensure 'go.uber.org/zap@<1.4.0'# git diffdiff --git a/depdemo/Gopkg.lock b/depdemo/Gopkg.lockindex fce53dc..3b17b9b 100644--- a/depdemo/Gopkg.lock+++ b/depdemo/Gopkg.lock@@ -16,12 +16,12 @@ [[projects]]   name = "go.uber.org/zap"   packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]-  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"-  version = "v1.4.0"+  revision = "6a4e056f2cc954cfec3581729e758909604b3f76"+  version = "v1.3.0" [solve-meta]   analyzer-name = "dep"   analyzer-version = 1-  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"+  inputs-digest = "3307cd7d5942d333c4263fddda66549ac802743402fe350c0403eb3657b33b0b"   solver-name = "gps-cdcl"   solver-version = 1

In this case, the version in Gopkg.lock does not meet the constraints in GOPKG.TOML. This also makes me more confused!

Third, DEP exploration

The above DEP uses basic workflows to fully meet the needs of daily package management. But for me who likes to find the solution, it is necessary to explore the behavior and principles behind Dep.

1. Two different results of DEP Init

We return to the initial state of Depdemo, the starting point: the moment the DEP metadata file has not been generated. In both cases, we perform DEP init separately:

• There's no go.uber.org/zap $GOPATH/src.

# dep init -vSearching GOPATH for projects...  Using master as constraint for direct dep github.com/beego/mux  Locking in master (626af65) for direct dep github.com/beego/muxFollowing dependencies were not found in GOPATH. Dep will use the most recent versions of these projects.  go.uber.org/zapRoot project is "github.com/bigwhite/experiments/depdemo" 1 transitively valid internal packages 2 external packages imported from 2 projects... ...# dep statusPROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USEDgithub.com/beego/mux  branch master  branch master  626af65   626af65  1go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7

• The existence of the/SRC under the $GOPATH go.uber.org/zap

# dep init -vSearching GOPATH for projects...  Using master as constraint for direct dep github.com/beego/mux  Locking in master (626af65) for direct dep github.com/beego/mux  Using master as constraint for direct dep go.uber.org/zap  Locking in master (b33459c) for direct dep go.uber.org/zap  Locking in master (908889c) for transitive dep go.uber.org/atomicRoot project is "github.com/bigwhite/experiments/depdemo" 1 transitively valid internal packages 2 external packages imported from 2 projects... ...# dep statusPROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USEDgithub.com/beego/mux  branch master  branch master  626af65   626af65  1go.uber.org/atomic    *              branch master  908889c   4e33664  1go.uber.org/zap       branch master  branch master  b33459c   b33459c  7

We do not know the similarities or differences between the results of the two cases. We only look at the Zap line in the two DEP status output:

go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7vs.go.uber.org/zap       branch master  branch master  b33459c   b33459c  7

DEP automatically analyzes two different results.

In the first case, we call DEP Init's network mode, where DEP finds no zap under local gopath, and DEP Init looks for ZAP on the network to upstream, and DEP will uses the most Recent versions of these projects ", or v1.4.0 version.

In the second case, we call DEP Init's Gopath mode, where Dep discovers that ZAP exists under local gopath, and DEP init determines "Using Master as constraint for direct DEP go.uber.org /zap ", that is, Master branch.

As for why Gopath mode, DEP Init chooses master, I guess because DEP thinks that since you have zap locally, it's quite possible that the stability of Zap Master is acceptable to you. In "dep:updated command spec", it seems that DEP Init intends to differentiate between the two modes of operation by adding a-gopath flag and using network mode as the default mode of operation. However, the DEP version I am using does not yet implement this feature, and the default mode of operation is still to Gopath mode first, and if no dependent package exists, network mode is implemented for that package.

As you can see from here, for DEP init output constraints, you'd better check to see if it's acceptable, or you can correct the output of DEP by using the "specify constraint" mentioned above.

2. DEP dependency package cache for the project

In the previous experiment, we found that, in the case of a local gopath/src without Zap, DEP would appear to be a direct zap get to the local vendor directory instead of get to GOPATH/SRC under Copy to vendor. What is the truth? DEP does not operate the GOPATH/SRC directory because it is shared. Dep leaves a "private plots" under $gopath/pkg/dep/sources for all cache dependencies that are downloaded from the network:

# ls-f $GOPATH/pkg/dep/sources/https---github.com-beego-mux/https---github.com-uber--go-atomic/            HTTPS---github.com-uber--go-zap/# ls-af/root/go/pkg/dep/sources/https---github.com-uber--go-zap./buffer/ Config_test.go field.go. Gitignore http_handler.go LICENSE.txt options.go s Ugar.go Writer.go.   /changelog.md contributing.md field_test.go glide.lock http_handler_test.go logger_bench_test.go Readme.md sugar_test.go writer_test.goarray.go check_license.sh* doc.go flag.go Glide . Yaml internal/logger.go. Readme.tmpl time.go zapcore/array_test.go Common_te       St.go encoder.go flag_test.go global.go level.go logger_test.go stacktrace.go         Time_test.go zapgrpc/benchmarks/config.go encoder_test.go. Git/global_test.go level_test.go        Makefile      Stacktrace_test.go. Travis.yml zaptest/ 

DEP is for a dependent package, so git requests are in this cache directory.

3, Vendor Flatten flattening

When go joins the vendor mechanism in 1.5, it is considered that there is a different version of the same dependent package in the "diamond-dependent" form. Let's see if DEP supports this. We have designed an experiment:

We set up a "diamond-shaped" test environment, Foo relies on a, b two packages, whereas A, b two packages rely on different versions of F (by declaring this constraint in a, B, gopkg.toml, see the callout in the figure).

Below is the main.go under the Foo project:

// foo/main.gopackage mainimport "bitbucket.org/bigwhite/b"import "bitbucket.org/bigwhite/a"func main() {    a.CallA()    b.CallB()}

Before we introduce DEP, let's Run the code:

$go run main.gocall A: master branch   --> call F:    call F: v1.1.0   --> call F endcall B: master branch   --> call F:    call F: v2.0.1   --> call F end

You can see the output of the same F package, because A, b depends on the different versions of F, so the output is different.

We have a DEP analysis of Foo to see what DEP has given us:

$DEP init-vsearching Gopath for projects ... Using Master as constraint for direct DEP bitbucket.org/bigwhite/a Locking in master (9122a5d) for direct DEP BITBUCKET.O rg/bigwhite/a Using Master as constraint for direct DEP bitbucket.org/bigwhite/b Locking in master (2415845) for direct  Dep bitbucket.org/bigwhite/b Locking in master (971460c) for transitive DEP Bitbucket.org/bigwhite/froot project is "Foo" 1 transitively Valid Internal packages 2 external packages imported from 2 projects ... No versions of bitbucket.org/bigwhite/b met Constraints:master:Could not introduce Bitbucket.org/bigwhite/b@master, a S it has a dependency in bitbucket.org/bigwhite/f with constraint ^2.0.0, which have no overlap with existing constraint ^1 .1.0 from Bitbucket.org/bigwhite/a@master v2.0.0:could not introduce bitbucket.org/bigwhite/b@v2.0.0, as it isn't all    Owed by constraint master from Project Foo. V1.0.0:could not introduce bitbucket.org/bigwhite/b@v1.0.0, as it's not allowed BY constraint master from Project Foo. Master:could not introduce Bitbucket.org/bigwhite/b@master, as it had a dependency on bitbucket.org/bigwhite/f with const Raint ^2.0.0, which have no overlap with existing constraint ^1.1.0 from Bitbucket.org/bigwhite/a@master

DEP Init failed to run. Because there is no intersection between the f@^1.1.0 and the B-dependent f@^2.0.0 two constraints, which cannot be reconciled, DEP cannot solve this dependency, so init failed!

But the reason behind the failure is that DEP is designed to require flatten vendor, which means that a project using DEP can have only one root vendor, so the vendor directory will be strip if the directly dependent or transitive package contains vendor. This way, if there are conflicting constraints in the dependency package, DEP init will fail.

Iv. Summary

Dep an important feature is to support the Semver 2.0 specification, but semver rules a lot, not here can be said clearly, you can go to the Semver official station to read the rules, or in NPM semver Calculator This site intuitively feel the changes brought about by semver rules.

The DEP test is over. From now on, DEP has entered the available stage, suggesting that conditional child shoes can actively use DEP and pre-test for DEP, uncover problems issue, and contribute to the rapid improvement of DEP.

The code for Depdemo is here; A, the code for the B,F package is here, here, and here.

V. References

• DEP FAQ
• DEP Roadmap
• DEP updated Command spec
• DEP Features
• The Saga of Go Dependency Management by Sam Boyer
• GPS for Implementors

Weibo: @tonybai_cn
Public Number: Iamtonybai
Github.com:https://github.com/bigwhite

, Bigwhite. All rights reserved.

Related Posts:

1. Understanding Go 1.5 Vendor
2. A few notable changes in Go 1.6
3. GODEP Support Go 1.5 vendor
4. CVS Primer
5. Advanced CVS