Maybe Go development can be more simple! Go + = Package Versioning

This is a creation in Article, where the information may have evolved or changed.

Comment

: There is no good plan for Go-dependent package versioning, although the community has been born with no more than 10 tools to solve the problem, but there has been no official support. This proposal is expected to go
In the next version of the official package version control, remove the Gopath dependency, but also introduced the concept of module, the true meaning of the re-compilation, is a big change

Compilation: Muse's Lover

It's time to add a package version control to Go!

Rather, we need to popularize the concept of the package version to the common vocabulary of Go developers and tools to accurately and consistently express what program code needs to be compiled, run, and parsed in subsequent communication. Similarly, the go command needs to tell the developer exactly which version of the package was used in the compilation.

Versioning allows us to implement recompilation. When I let you try out the latest version of my program, I know that you are not only getting the code for my latest program, but also the same version of the package that my code relies on to compile the exact binary package.

Versioning also allows us to maintain the same compilation at different stages, even if our dependency packages may have new versions, and the go command will not use the new version of the package as long as our configuration is not allowed to be used.

While adding versioning is a must, we cannot afford to lose the good features of the GO command line: simple, efficient and easy to understand. At the moment, many programmers still don't pay much attention to version control, mainly because there is no problem in most cases. Imagine that if we have a reasonable pattern of design and default configuration, so that developers do not need to pay much attention to version control things, the program can still work very well, and less impact on existing projects, release the new version is simple enough, and even the daily development can ignore the version control of the work, this version control mode is what we want.

In short, versioning is a must, but it should be transparent enough not to break down go get its functionality. This article explores a proposal that fully implements these, and provides a prototype demo that can now be used to lay the groundwork for integration into go commands. I'm going to discuss in this article what needs to be done in the process of production and what is not, based on this discussion, I will make further adjustments to the proposal and the prototype, and submit an official proposal, integrated into the Go 1.11 version as an optional feature.

This proposal retains go get the best part, adds duplicate builds, uses semantic versioning, discards, discards the vendor dependency on Foundation project creation, GOPATH and provides a smooth migration of old projects in a way that is still at an early stage, if there is a problem in the details We will fix it before the go major release

Background

Before we discuss this proposal, let's look at the present situation. It may be a long story, but the lessons of history have important implications for the present, and let us be clear about what the proposal has changed. If you're bored, skip to the proposal or go to the prototype demo

Makefiles, Goinstall, and go get

In November 2009, Go released a first-generation version with a compiler, a linker, and some built-in libraries, at which point you had to run 6g and 6l compile and link your program, with some simple makefiles. In most cases, gobuild a single package program can be compiled with a simple wrapper and the corresponding makefile will be generated. At the time, there was no proper way to share the code with others. Although many features are not available, they are still released, and go plans to put some of the remaining features into the community.

In February 2010, Goinstall appeared, a new 0 configuration command line, primarily used to download packages from the source management library (like BitBucket and GitHub). Goinstallintroduced the path conventions that have been popularized today in go developers, because no code followed this Convention at the time, goinstall initially only on the import of standard libraries, but developers quickly migrated their own naming conventions to the uniform conventions that we know today, and these released go Packages gradually formed a coherent ecological system.

GoinstallIt also discards the makefiles, eliminating the complexity of the user's build configuration. Although it is occasionally inconvenient for the package author to generate code every time it is built, this simplification is important to the user of the kit: the user does not have to worry about inconsistencies in the compilation of the Toolset in the installation of the toolkit and the compile of the assembly author. Simplification is meaningful for a tool. There is a certain necessity for makefile to compile the package in step, and how to make the same package use different tools (like go vet or code completion) in reverse engineering is makefile difficult to do in this respect. Even if the compilation dependencies are properly maintained, it can be difficult for any makefiles to re-compile if necessary. Although some people think that removing makefiles loses flexibility, but looking back, the benefits are much greater than these inconveniences.

In December 2011, as part of the pre-release version of Go 1, we introduced the use of the GO command go get to replace goinstall .

Overall, go get It's transformative, it allows Go developers to share code and build with each other, and the tools isolate the details of the go command-compilation system, but the go get lack of versioning is actually goinstall The first discussion was clearly aware of the need for versioning-related features. Unfortunately, at least at the time our go team was not sure how to do it. When go get requires a package, always download the latest copy from a remote version control system like Git or Mercurial, and the lack of package versioning has at least two major flaws.

Versioning and API stability

First, a major flaw in go get when there is no version control is that the given update is not known whether the user expects it.

In November 2013, Go 1.2 added a FAQ about basic recommendations for package versioning:

packages that are publicly released should be backwards compatible. Go 1 Compatibility guidelines
Provides a good reference: Do not delete the exported naming and encourage the use of conforming semantics (composite
Literals) tag name version and so on. If you need a different feature, add a new name instead of modifying the old name, if you need a completely separate feature, create a new under the new import path

Package

In March 2014, Gustavo Niemeyer created gopkg.in, advocating "Go language stability API". This web site is actually a redirect that senses the GitHub version change, and you can gopkg.in/yaml.v1 gopkg.in/yaml.v2 point to the different commit versions of the Git libraries (and possibly the different branches) by using such an import method. Following this line of thinking, the ability to rely on the library has changed significantly after you can import the previous V1 version of the path as a fallback, and then create a new version of V2, through the V2 import path to introduce a completely different API.

In August 2015, Dave Cheney proposed a semantic versioning proposal that sparked an interesting discussion in the next few months, and everyone thought that semantic versioning was a good idea, but no one knew what to do next: What kind of tools should the semantic version use?

Any discussion of semantic versioning will inevitably be countered with the Hiram rule:

When an API has enough users, it doesn't matter what you promise in the contract, and all the behavior observed in your system will be directly dependent on the user.
Although the Hiram rule is correct in experience, semantic versioning is still an effective way to establish a reasonable expectation relationship between different releases. In general, upgrading from 1.2.3 to 1.2.4 should not destroy your original code, but upgrading from 1.2.3 to 2.0.2 may break the original code. If your version upgrade to 1.2.4 has a problem, then the author will generally be based on the bug report in 1.2.5 repair, if your code to upgrade to 2.0.0 a problem, then there may be a major upgrade deliberately.

Vendoring and Repeatable builds

go getThe second major flaw with no versioning concept is the idea that you might not be able to implement recompilation. You have no way to confirm that the user of your program compiles and your compile time relies on the same version of the package, in November 2013, the Go 1.2 FAQ also added the following basic recommendations:

If you are using an externally supplied package to worry that it will change unexpectedly, the simplest way is to copy it to your local library (which is what Google uses internally) and mark it as a local repository and put it into a new import path, such as "http://original.com/pkg" copying to "http://you.com/external/original.com/pkg" . Keith
Rarick's Goven is an automated tool to implement this functionality

GovenIt was Keith Rarick, released in March 2012, which copies the packages you rely on to your local repository and updates all the import paths to the new local path, and modifying the source code's dependencies in this way is valid for compilation, but there are some problems, This modification makes it difficult for local packages to compare changes to new replicas to merge the required updates.

In September 2013, Keith released godep , "a new tool to freeze package dependencies." The godep most important promotion is the addition of what we now know-- Go vendoring no need to modify the source file to copy the dependencies to the project--no tool chain support, a way to set the GOPATH implementation

In October 2014, Keith suggested adding "external package" concept support to the Go tool chain so that the tool could better analyze the project according to the contract. At that time, there were also a lot of tools similar to GODEP, Matt Farina wrote an article "Glide in the Sea of Go package Managers" compared GODEP and a lot of later tools, the most prominent of which is glide .

In April 2015, Dave Cheney introduced the GB, a "project-based build tool ... The source code in the vendor is repeatable, and there is no need to re-import, ( gb another goal is to avoid placing the required codes in a specially specified GOPATH directory, which is not a good development process for many developers).

That spring, Jason Buberel The Go Package management tool to understand how to integrate the results of these different people's efforts to avoid duplication and wasted work. His investigation made it clear to the go team that the GO command needed to support vendoring directly in case the package did not re-import. At the same time, Daniel Theophanes began to develop a file format specification to describe the exact source and version information of the code in the vendor directory. In June 2015, we accepted Keith's proposal vendor as the Go 1.5 experimental feature, which is optional in Go 1.5 and is the default feature in Go 1.6. We encourage all vendoring tool authors and Daniel to work together to develop a document specification in a unified metadata format.

vendoringincorporating concepts into the Go Tool chain allows program analysis tools like go vet to better understand engineering. So far there are dozens of Go package management tools or vendoring tools to manage the vendor directory. On the other hand, because these tools use different metadata-based file format specifications, they cannot easily share dependent information.

More importantly, it is vendoring not a complete solution, it simply provides a recompile implementation without addressing the issue of package versioning, and it does not help engineering to understand the version of the package to decide which version of the package to use. glide dep The package manager, like and such, is vendor implicitly introducing versioning concepts into the go compilation by some kind of setting of the directory, and does not require direct tool chain support. In fact, many of the tools in the go ecosystem do not recognize versions, and it is clear that go needs to provide a direct tool chain support for the package version.

An experiment of official package management

On the 2016 Gophercon, a group of Gophers who were interested in go gathered around hack day to have a discussion about Go package management, one of which was to set up a committee and a package management Advisory group with the goal of creating a new Go package management tool, The vision is to unify some of the existing tools that are implemented in a way that still works through the vendor directory. Initiated by Peter Bourgon and a member of the committee Andrew Gerrand, Ed Muller, Jessie Frazelle, and Sam Boyer jointly drafted a specification for DEP with Sam-led and background information see Sam in 2 016 February Published article "So you want to write a package manager" and December 2016 article "The Saga of Go Dependency Management" as well as July 2017 Gophercon Share "The New Era of Go package Management."

DepThere are multiple uses: It is an improvement based on current best practices and an important step towards a successful solution, as well as an "official experiment," which helps us gain a deeper understanding of what we do and what we do not do for go developers. But it dep is not the final prototype of the GO Command Integration Pack version control. It explores the space of design in a way that is powerful and flexible, and can be said to play a makefiles function when we compile the Go program, and once we have a deep understanding of its design ideas and focus on several key functions that must be supported, you will find that the design will remove the go Many of the other features in the ecosystem reduce the cost of expressing power, while mandatory conventions make the go code look more uniform and understandable, making it easier to build tools.

Let's look at the next dep stage of the goal: complete the go command to integrate the final prototype of the first draft, similar to goinstall the package management, this prototype is a separate command we call vgo it, you can think of it as a support package version control of the Go command. This is a new experimental feature, and as it was introduced goinstall , some code and engineering has been supported vgo , and some other projects need to be adapted. As with removal makefiles , some control and presentation related things are removed, simplifying the system to reduce user costs.

vgoExperiment does not mean that we will stop support for DEP, and we will continue to ensure that dep the path that is available until the go command integration is determined, implemented, and available. Of course, we also ensure that we can dep move from smooth to go command integration, if the project is not used dep (note that GODEP and glide have stopped updating, it is recommended to migrate to DEP) then you can migrate directly tovgo

Proposal

There are four steps in the proposal to go add version control to a command. The first is to be compatible with the Go FAQ and gopkg.in the import rules in, that is, to establish an expectation that the new version of the package import path should be backward compatible with the old version. Second, a simple new algorithm (known as the minimum version selection) is used to filter which package version is used at compile time. Thirdly, by introducing the concept of Go modules, the Go module is a set of dependencies that contain a single version of the package and declare the minimum version they require. Finally, define how these changes are integrated into the GO command, so the basic workflow of the Go command from now on will not change much. Next, we'll look at it in detail, and this week I'll do a more detailed introduction to other articles.

Import compatibility rules

The biggest pain in the package management system is resolving compatibility issues. For example, most systems where package B claims require a package D version of 6 or higher, then package C declares that the required package D version is 2,3 and 4, but not higher than version 5. If you are writing package A and you want to introduce packages B and C at the same time, then you are unlucky: there is no separate D version for B and C to choose to compile into a. B and C are reasonable, and you can't change it, so you're stuck.

To prevent the leader from designing a system that causes existing large programs to fail to compile, the proposal requires the package author to follow the following import compatibility guidelines:

If an old package and a new package have the same import path, the new package must be backwards compatible with the old package
This rule is a reaffirmation of the previous Go FAQ, citing the last of the FAQ: "If you need to change completely, create a package with a new import path." Developers want to be able to express such a change through a semantic version, so we've added semantic versioning to our proposal. Specifically, major version 2 and newer versions can be differentiated by including version information in the path, such as:

import "github.com/go-yaml/yaml/v2"

Created the v2.0.0 version, which means a major change in semantic versioning, followed by the import compatibility principle to create a package for a new import path. Because each major version has a different import path, the given go executable may contain either of the major versions, which is exactly what we expected.

The package author follows the import compatibility principle, which allows us to reduce the adaptation effort and make the system simpler while also reducing the fragmentation of the package ecology. Of course, in fact, despite the author's best efforts to do, the update will inevitably appear to undermine the user's use of the situation. Therefore, it is important to use an upgrade mechanism that is not frequently upgraded, which is what we will talk about next.

Minimum version selection

Almost all of the package management include dep and cargo are built with the latest package version, based on two important factors, I think this is a wrong convention. First, the "Latest available version" is likely to cause changes due to external events, such as the release of a new version. Maybe someone in the package you're relying on tonight will release a new version, and the next morning you'll probably have a different result when you compile. Second, in order to override this default convention, developers spend a lot of time telling the package Manager which version of the package is not used.

In the proposal we used a different approach, called the minimum version selection. Each package is built with the oldest available version by default, which makes the compilation of yesterday and today not change, because you will not be releasing an older version today. Better yet, developers simply tell the package Manager which version is the least available, and the package manager can quickly decide which version is available. We call it the minimum version. The choice is on the one hand because we choose the smallest version, on the other hand, because it is minimized for the entire system and avoids the complexity of the existing system.

The minimum version selection for the module specifies the minimum version requirements for its dependent modules, which provides a good choice for subsequent upgrades and demotion operations. It can also be compiled by excluding the dependency on the specified version or by specifying a special version dependency.

Minimum version selection A repeatable build is done by default without locking the file.

The minimum version selection is the key to import compatibility. Users will not say: "No, the version is too new", more cases are faced with "no, the version is too old", in this case the solution is very clear: Upgrade the new version is OK.

Go Modules

The Go module is a collection of packages that share an import path prefix, which is what we call the module path. A module is a version-controlled unit, and the version of the module is represented by a semantic version string, and when used in development Git , the developer Git defines a new semantic version by adding a new tag to the module's repository. Although it is strongly recommended to use the semantic version, it also supports pointing to a specific commit.

The module is defined in a go.mod new file called, which contains the minimum version of the package that the module depends on. Here's a simple go.mod file:

// My hello, world.module "rsc.io/hello"require ("golang.org/x/text" v0.0.0-20180208041248-4e4a3210bb54"rsc.io/quote" v1.5.2)

This file defines a module through the path identifier rsc.io/hello , which itself also relies on two other modules: golang.org/x/text and rsc.io/quote , this module compiles itself by using the version of the dependency list specified in the Go.mod file. For a more up-level compilation, other places that import this module will be compiled with a newer version of it.

The best way for a package publisher to make a semantic tag release is to vgo encourage it by hitting the tag's version number instead of any commit version. rsc.io/quotethe module uses a tag version of the method, and the golang.org/x/text module does not provide a tag version. For unnamed commits, v0.0.0-yyyymmddhhmmss-commit represents a commit of a specified date, in semantic versioning, the string v0.0.0 represents the pre-issued version number, and Yyyymmddhhmmss-commit represents the identifier for the pre-release version.

In addition to specifying the required dependent versions, the go.mod files can also implement the excluded and replaced versions mentioned in the previous section, but these only work when the module is compiled directly, not when the module is compiled as part of the overall project, and can be viewed in detail in this example (examples).

GoinstallAnd the old go get way of downloading code directly through versioning tools like Git and HG, there are many problems, including fragmentation: Users bzr can't download code hosted in the Bazaar repository if they don't. In contrast, the module is the way to download the ZIP package over HTTP. Previously, a package that encountered special needs was go get downloaded through a version-controlled command-line tool to mainstream code hosting sites, and now vgo downloads the required packages directly from the API provided by the website.

Module unified through the form of a ZIP package can make the download protocol simpler, the company or individual can be considered for any reason (security or want to cache a copy to prevent the source from being deleted) to do the download agent, use the agent to ensure availability and by go.mod defining what code needs to be used, vendor The directory is no longer needed.

Go command

The go command must be updated to use the module feature. An important change is the common build command, like, gobuild , go install go run and go test will need to resolve the corresponding dependency according to the specified requirements, in the new module golang.org/x/text can only be used in the go source code to import and compile, do not need to pay attention to the version of the issue alone.

However, the most important change is to end the GOPATH setup as a Go Code workspace, Because the Go.mod file contains the full module path and defines each dependent version that is used, the directory containing the Go.mod file can be considered the root of a directory tree, which acts on its own workspace and is isolated from other similar directories. Now all you have to do git clone cd is get the code straight and you don't need GOPATH it anymore.

Next plan?

I also published an article "a tour of versioned Go", mainly about the use of vgo some feelings. With this article you can find out how to download and experience now vgo , and I'll be releasing a few more articles this week to supplement some of the things I skipped in this article. I hope you can make some suggestions for this article, I will try to see the Go subreddit and golang-nuts email feedback. In Friday I will publish an end of the FAQ and I will submit a formal go proposal next week.

Try the VGo bar and start tagging the version with tag in your repository to create and check in the Go.mod file. Note If your repository has an empty go.mod but exists,,, dep , glide glock godep godeps govend govendor or gvt configuration file, it vgo will populate the Go.mod file with them.

Original: Https://research.swtch.com/vg

...