The New Era of Go package Management

This is a creation in Article, where the information may have evolved or changed.

Liveblog by Alan Shreve (@inconshreveable)

Update:slides for this talk has been postedhere.

Author

Presentation by Sam Boyer @sdboyer.

Sam fell into software engineering by accident and decided to stick around. He loves complex, ecosystem-class problems, cares deeply about building healhty communities, and are always looking for s to bring technology's enormous potential to bear in the world ' s critical problems. Nowadays, he ' s a Site reliability Engineer at Vividcortex.

The new era of Go package management

The new era of the package Management-we ' ve all been waiting for it. We ' re on the road towards official tooling for package management. It's going to is a real sea change for Go development.

What's the package management

Let's all being on the same page about what's the management is:

Step 1. Write Code

Step 2. Pull in other people ' s code

Step 3. PAIN

Package management are primarily about:

• Reproducibility
• Managing updates from Upstream

The package and dependency management are technically different, but we'll talk about them interchangeably here.

Package management:we don ' t want to deal with it, isn't fun.

Package management are more than just tech. We ' re talking about building tools, that interact and other developers. Not just a technical problem. Tools-Social Systems.

Just like how telephones has changed our social systems, dependency management systems, and our social systems. Dependency Management makes boring social systems but they is still important because they is the mediators of Coordinat Ion between people writing software.

Analogy with pidgin languages:a Pidgin are A new language that emerges when both unintelligible languages meet. They usually emerged around trade routes. No one speaks them natively (pidgin language by definition). Pidgins is a great tool for culture. Dependency Management in Go is like a pidgin language. There is no official tooling, similar to the lack of official language when Pidgins emerged. Parties trying to coordinate dependencies used or built whatever tools best suited them, but there is no overall comm Unity Buy-in.

Community buy-in and a standard tool is exciting because the quality of software would be better with an official Coordinat Ion tool or ' language. ' We ' re looking forward to more than a new era of the package management, we ' re looking forward to a new era of Go.

The challenges of GO package management

Let's talk about the fundamental problems on Go package management.

• If packages want to rely on different version of a shared dependency
• No reproducibility, go get only cares about tip
• The official tooling only caring on tip means as a community we don ' t care about releases
• Since we don ' t care about releases we don't have a sane by the think about updates

The Go community have worked around it with discipline:

• Try not to use dependencies unless they is absolutely necessary
• Maintainers make great efforts to don't change APIs

History of the Go package management

Timeline

• 2013:godep
• 2014:glide gopkg.in
• 2015:GB, Govendor

Tools

• Godep:originally about reproducibility, restore/save your Gopath
• Gopkgin.in:putting version information in the URL
• Gom, glide:modeled after tools on other languages
• The Vendor directory

Vendor

• 2015:vendor added, off by default
• 2016:vendor on by default
• 2017:vendor always On

Vendor is isn't without its own problems:

• Nested Vendor directory problems.
• Creates type incompatibilities.
• Causes problems with global state of packages or init () packages that is duplicate.

These problems became very prominent when vendor is enabled by default.

State of the Go package management in 2016

Too often, when the trying to create a unifying standard, we just create another competing standard. It's a testament to the awesome Go community that we're not creating a 15th standard but one that we'll be adopting to Gether.

So we started an initiative to put together a group of Go community leaders to work on the package management. The task is to take community feedback, write design docs, and understand competing concerns with respect to package mana Gement.

Constraints of the package Management working Group

• Build a working prototype for who dependency management in Go should become
• Try not to add new rules or problems
• Make migration from existing systems as minimal and painless as possible
• Most important: Obtain community buy-in for the vision

Github.com/golang/dep

This is the result. We call it ' the official experiment '. Those words is carefully chosen.

There is a huge amount of community participation in the mailing list while it is developed.

DEP Fundamentals

• Borrows from others, but tailored to go
• Imports are Queen
• Two-file System:gopkg.toml/gopkg.lock (gives us reproducibility)
• project-oriented
• Semver tagging
• Vendor-centric, built around working with the Vendor directory

The three commands of DEP

• Status:what is the current state of the project on disk?
• Init:create a new project, automated logic to port older formats like Glide and GODEP to the DEP system
• Ensure:the most important command

DEP ensure

Ensure command keeps state in sync. Ensure is about making sure the states be in sync according to the defined rules. The whole DEP tool is designed around state resolution instead of requiring, the user to issue many imperative commands. This means it is a very opinionated tool. So much so, we have our own mascot.

The ensure model is leaking to other communities ' DEP tools (like NPM).

DEP moving forward

DEP is at the center of the where we're right now, but it's not the end goal. The end goal is to move this functionality into the official toolchain. Everything is being designed with the intent so it we would like it to eventually become part of the Go toolchain.

In this-to-do DEP is a stepping stone. This was the first time we ' re looking at eventually bringing a huge amount of Third-party community code into the official Toolchain. This is uncharted territory and still a very big experiment.

One key insight from Dep:what we ' re discussing is a third space beyond vendor & Gopath. A third space would has a versioned set of packages on disk.

DEP TODO List

• Multi Project Workflow. How does I work on the projects locally that depend in each other?
• Semver sugggestion tool:example from Elm. Codifying the Go 1 guarantee about "re allowed to the change
• Plus a lot more

Help DEP needs from the community

• Please, please tag your projects with Semver
• Convert your projects to Dep. It ' s ready for production. Guaranteed backwards compatibility of the file formats
• Scripting on top of it:hold off for a little bit because the CLI options could change some
• Come contribute to the DEP project!

The MIT License: "FITNESS for A particular PURPOSE" are not guarantee. We ' re here because that ' s true. That's true of all the code and its a inherent complexity in working in software. DEP management can ' t remove that problem, but it matters because it can make that complexity assailable. The DEP complexity burden is something this good tools can help you tame.

Dep lives to Theatgoal of managing the complexity.

Huge Thank you's to all the maintainers and contributors to all Go dependency management tools. And thank the Go team for their trust and especially to the community for helping make this project a reality.

About Sourcegraph

Sourcegraph is a code search and browsing tool with code intelligence (Go-to-definition and Find-references, even across R epositories). Try it out on GOLANG/DEP.