Understanding Go 1.5 Vendor

This is a creation in Article, where the information may have evolved or changed.

Go 1.5 (currently the latest version of GO1.5BETA3) added a experimental feature: vendor/. This feature is not a formal feature of Go 1.5, but it is an important attempt by go authors to manage the package dependencies that go is being criticized. The current information on the go vendor mechanism is limited, and the main ones include the following:

1, Russ Cox on the Golang-dev group on a named "Proposal:external packages" topic on.
2, Go 1.5beta release after Russ Cox according to the above topic finishing a doc.
3, medium.com on an article called "Go 1.5 vendor/experiment".

However, since the Go 1.5 stable version has not yet been released (the latest news is released in mid-2015.8 months), it is estimated that the real vendor repo is not yet available. But since it is the official go solution, the possibility of subsequent changes from Expreimental to official is very large (Russ's preliminary plan: If the test goes well, the default version of 1.6 go15vendorexperiment= "1" ; 1.7 will remove the go15vendorexperiment environment variable). So for the Gophers, it is necessary to figure out vendor. This article and everyone together to understand the next vendor this new feature.

First, the origin of vendor

Go third-party package dependency and management problems for a long-standing, well-known solutions have GODEP, GB and so on. The Go team had done a long-time research on Golang-dev group before the launch of vendor, and eventually Russ Cox improved on the proposal of Keith Rarick, forming the vendor in Go 1.5.

Russ Cox based on the results of the previous research, this paper gives the mass opinion base of vendor mechanism:
– No rewrite Gopath
–go tool to solve
–go Get Compatible
– Can reproduce building process

and gives the "4 lines" interpretation of the vendor mechanism:

If There is a source directory D/vendor, then, when compiling a source file within the subtree rooted at D, import "P" is Interpreted as import "d/vendor/p" if that exists.

When there is multiple possible resolutions,the most specific (longest) path wins.

The short form must always be used:no import path can contain "/vendor/" explicitly.

Import comments is ignored in vendored packages.

This four-line interpretation has aroused strong discussion in group, and the short-and-small is a very different understanding. We understand each of the following examples.

II. Basic sample of Vendor

The first article in Russ Cox's interpretation is the basis of the vendor mechanism. The rough understanding is that if you have a directory structure like this:

d/
vendor/
p/
P.go
mypkg/
Main.go

If there is "import P" in Mypkg/main.go, then this p will be parsed by the go tool to "d/vendor/p" instead of $gopath/src/p.

Now we're going to replicate this example, and we'll create the above directory structure under Go15-vendor-examples/src/basic (where Go15-vendor-examples is the Gopath path):

$ls-R
d/

./d:
mypkg/vendor/

./D/MYPKG:
Main.go

./d/vendor:
p/

./d/vendor/p:
P.go

Where the Main.go code is as follows:

Main.go
Package Main

Import "P"

Func Main () {
P.P ()
}

The P.go code is as follows:

P.go
Package P

Import "FMT"

Func P () {
Fmt. Println ("P in d/vendor/p")
}

When vendor is not turned on, we compile d/mypkg/main.go to get the following error result:

$ go Build main.go
Main.go:3:8:cannot Find Package "P" in any of:
/users/tony/.bin/go15beta3/src/p (from $GOROOT)
/users/tony/opensource/github.com/experiments/go15-vendor-examples/src/p (from $GOPATH)

The reason for the error is obvious: The go compiler could not find the P under package P,d/vendor at this time.

At this point the Vendor:export go15vendorexperiment=1 is turned on, and we will compile it again:
$go Run Main.go
P in d/vendor/p

The Go tool, which opens the vendor mechanism, finds the package p under D/vendor.

In other words, after you have vendor, your project relies on a third-party package that is all under vendor/. This will download the third-party package at the same time, so that your project can be downloaded there without having to rely on the target environment to compile and pass (reproduce the building process).

Three, nested vendor

So here's the problem! If the vendor directory is also included in the third-party package in vendor, how does the Go tool choose third-party packages? Let's take a look at the following directory structure (go15-vendor-examples/src/embeded):

d/
vendor/
p/
P.go
q/
Q.go
vendor/
p/
P.go
mypkg/
Main.go

Nested vendor structures appear in the embeded directory: the Main.go-dependent Q package itself has a vendor directory with a P-packet in it, so we have two P-packages. Which P-Pack does the Go tool choose? Obviously in order to verify some conclusions, we have to change the source file:

The D/vendor/p/p.go code does not change.

D/vendor/q/q.go
Package Q

Import (
"FMT"
"P"
)

Func Q () {
Fmt. Println ("Q in d/vendor/q")
P.P ()
}

D/vendor/q/vendor/p/p.go
Package P

Import "FMT"

Func P () {
Fmt. Println ("P in d/vendor/q/vendor/p")
}

Mypkg/main.go
Package Main

Import (
"P"
"Q"
)

Func Main () {
P.P ()
Fmt. Println ("")
Q.Q ()
}

The catalogue and the Code are finished, and we come to the moment of witnessing miracles! We execute the main.go:

$go Run Main.go
P in d/vendor/p

Q in d/vendor/q
P in d/vendor/q/vendor/p

It can be seen that the final reference in Main.go is d/vendor/p, and Q. Q () in the called P. P () is the implementation of the D/VENDOR/Q/VENDOR/P package. How exactly does the go tool choose a package in the case of nested vendor? We go back to Russ Cox's second article on vendor interpretation:

When there is multiple possible resolutions,the most specific (longest) path wins.

This sentence is very brief, but it draws a huge controversy. "Longest path wins" puzzles people. If only the literal meaning, the above Main.go's execution result should be:

P in d/vendor/q/vendor/p

Q in d/vendor/q
P in d/vendor/q/vendor/p

The d/vendor/q/vendor/p can be longer than the d/vendor/p path, but the Go tool obviously does not. How the hell did it do that? Talk was cheap, show you the code. Let's take a cursory look at the implementation code for the Go tool:

In $goroot/src/cmd/go/pkg.go there is a method Vendoredimportpath, which is widely used in the Go tool:

Vendoredimportpath returns the expansion of the path when it appears in parent.
If parent is x/y/z, then path might expand to X/y/z/vendor/path, X/y/vendor/path,
X/vendor/path, Vendor/path, or else stay x/y/z if none of those exist.
Vendoredimportpath returns the expanded path or, if no expansion is found, the original.
If No expansion is found, Vendoredimportpath also returns a list of vendor directories
It searched along the "the", to-help prepare a useful error message should path turn
Out isn't to exist.
Func Vendoredimportpath (parent *package, path string) (found string, searched []string)

The doc of this method is very clear, this method returns all possible vendor path, taking Parentpath as the example of x/y/z:

When X/y/z is entered as a Parentpath, the returned Vendorpath includes:

X/y/z/vendor/path
X/y/vendor/path
X/vendor/path
Vendor/path

This is not very intuitive, we combine our embeded vendor example to illustrate why the result is like the above! How the Go tool is resolve P package! We are emulating the go tool to compile the Main.go code (vendor is now turned on).

The Go tool compiles the P package first, according to the GO program's packages init sequence. How to find the P-bag? At this point the compiled object is D/mypkg/main.go, so the parent = d/mypkg, after Vendordimportpath processing, the possible vendor path is:

D/mypkg/vendor
D/vendor

But only d/vendor/under the P package, so go tool will p packet resolve to d/vendor/p, so the following p. P () will output:
P in d/vendor/p

The Q package is then initialized. Similar to P, the Go tool compiles the Main.go code, at which point the compiled object is D/mypkg/main.go, and then the parent = d/mypkg, after Vendordimportpath processing, the possible vendor path is:

D/mypkg/vendor
D/vendor

But only d/vendor/under the Q package, and then go tool will be Q packet resolve to d/vendor/q, because the Q package itself also relies on p packet, so go tool continues to Q in the P packet depends on the choice, the Go tool's compiler object becomes the d/vendor/q /q.go,parent = d/vendor/q, so after vendordimportpath processing, the possible vendor path is:

D/vendor/q/vendor
D/vendor/vendor
D/vendor

The paths that exist for P packets include:

d/vendor/q/vendor/p
d/vendor/p

At this time according to Russ Cox interpretation 2:choose longest, so go tool chose d/vendor/q/vendor/p, so Q. The P in Q (). The content of the P () output is:
"P in d/vendor/q/vendor/p"

If the directory structure is complex enough, this resolve process is quite cumbersome, but it is still possible to analyze the correct package according to this idea.

In addition, the Vendoredimportpath incoming parent x/y/z is not an absolute path, but a path relative to $GOPATH/SRC.

BTW, the above test sample code can be downloaded here.

Iv. Articles III and fourth

The most difficult to understand the second has been pass, the remaining two is better understood.

The short form must always be used:no import path can contain "/vendor/" explicitly.

This is to say that you do not care in the source code vendor the existence of this path, how to import the package on how to import, do not appear import "d/vendor/p" situation. Vendor is implicitly handled by the Go tool.

Import comments is ignored in vendored packages.

Go 1.4 introduces the canonical imports mechanism, such as:

Package PDF//import "Rsc.io/pdf"

If you quoted a PDF that is not from rsc.io/pdf, the compiler will make an error. However, due to the existence of the vendor mechanism, the Go tool does not verify whether the vendor package's import path is consistent with the canonical import paths.

V. Questions

According to the analysis in section three, the resolving process for packages in vendor is similar to a recursive (recursive) process.

P in Main.go uses d/vendor/p, and p in Q.go uses d/vendor/q/vendor/p, so there is a problem: there are two versions of P-packages in a project, which may not be a problem, and may also be the source of the problem. But for now, there seems to be no better way from the Go tool's point of view. Russ Cox expects everyone to design a good layout, as Lib's package does not carry vendor better.

All of the vendor in this project are centered on the top-level vendor. Just like this:

d/
vendor/
q/
p/
... ...
Mypkg1
Main.go
Mypkg2
Main.go
... ...

In addition, go vendor does not support version management of third-party packages, and there are no files like GODEP Godeps.json to store package meta-information. However, there have been third-party vendor specs on GitHub, before go team Brad Fizpatrick also in Golang-dev to solicit a similar scheme, do not know whether the future vendor will support.

Vi. Vendor vs. internal

In Golang-dev it was mentioned that vendor,internal seemed useless. This is clearly confusing the problems that internal and vendor have to solve.

Internal Name Incredibles: Internal package, not visible to all source files. Vendor is to store and manage external dependency packages, more like external, where the packages are copied from outside, and all source files within the project can be in the import vendor package. In addition, internal in the 1.4 version has been added to the go core, it is not easy to remove, although so far we have not been able to personally understand the role of internal package.

In the article "go 1.5 notable changes" I mentioned that go 1.5 beta1 seems "not supported" Internal,beta3 released, I tried to see if beta3 support internal package.

The result is beta3, build still does not error. But go List-json will prompt the error:
"Depserrors": [
{
"Importstack": [
"Otherpkg",
"Mypkg/internal/foo"
],
"Pos": "",
"ERR": "Use of the internal package not allowed"
}
]

Is it really going to be the final go 1.5 release that will make the internal package work?

Bigwhite. All rights reserved.