Announcing dep v0.5.0

July 25, 2018

v0.5.0 of dep has been released!

The big theme of this release is performance improvements. dep was designed for safety from the outset, because we knew that foundation would let us speed things up later. Now we have!

NOTE: your whole team will need to update at once to this new release, as it results in changes to the structure of Gopkg.lock that older versions of dep won't know how to work with.

Performance Improvements

There are two big aspects to the performance improvements: source metadata caching, and vendor verification.

Source metadata caching is an experimental feature that caches the result of all the parsing and code-backed analysis dep does as part of the solving process: reading in your dependencies' Gopkg.toml files, parsing the .go files for import statements, etc. All that work, and the git checkout necessary to put code on disk to analyze, is what made the solver plod along in the past.

With the caching enabled (managed by the env var DEPCACHEAGE), any combination of version and project that was already visited is retrieved from a persistent cache. Time per solving step drops to the (sub-)millisecond range; previously it was on the order of hundreds of milliseconds or seconds.

Vendor verification is the notion that Gopkg.lock should contain enough information to be able to verify whether the current contents of vendor/ are exactly as they should be, including whatever pruning options you've set. We've now done this, by adding the digest and pruneopts fields to each [[project]] stanza in Gopkg.lock.

The performance impact of all this is that it is no longer necessary for dep to rewrite the entirety of vendor/ on every dep ensure run. Instead, dep selectively writes out or removes only the files necessary to bring vendor/ back in line with Gopkg.lock. With -v, it'll also tell you why change was made:

# Bringing vendor into sync
(1/4) Wrote github.com/eapache/go-resiliency@v1.1.0: version changed (was v1.0.0)
(2/4) Wrote github.com/gregjones/httpcache@master: revision changed (2bcd89a174 -> 9cad4c3443)
(3/4) Wrote github.com/prometheus/common@master: prune options changed (UT -> NUT)
(4/4) Removed unused project github.com/kr/pretty

While the improvements affect different workflows in different ways, a representative dep ensure -v run (including both a solve and updating vendor/) for CockroachDB dropped from 120s to 4s in local benchmarking.

Improved feedback

Vendor verification has implications beyond just performance. With it complete, we fixed dep's final blind spot on whether all of the dependency-relevant information in your project - imports in code, Gopkg.toml, Gopkg.lock, and vendor/ - are in sync. That enables not only the granular feedback about vendor/ changes above, but it also lets us tell you exactly what changed in your project that pushed it out of sync, causing a solve.

dep informed you of this in the past, but it was kinda useless:

$ dep ensure -update -v
Warning: Gopkg.lock is out of sync with Gopkg.toml or the project's imports.

Not very helpful.

Now, though, if dep ensure -v sees your project is out of sync in a way that entails re-solving the graph, it will tell you exactly why:

$ dep ensure -v
# Gopkg.lock is out of sync
github.com/kr/pretty: imported or required, but missing from Gopkg.lock's input-imports
github.com/aws-sdk-go/aws/awserr: in Gopkg.lock's input-imports, but neither imported nor required
github.com/pkg/errors@v0.7.0: not allowed by constraint ^0.8.0

Of course, what if you just want to know what's out of sync, without actually changing anything? We have a new subcommand for that!

`dep check`

This release introduces a new subcommand, dep check, which reports all the ways that your project is out of sync. This includes the output of dep ensure -v, but also looks for any issues in vendor:

$ dep check
# Gopkg.lock is out of sync
github.com/kr/pretty: imported or required, but missing from Gopkg.lock's input-imports
github.com/aws-sdk-go/aws/awserr: in Gopkg.lock's input-imports, but neither imported nor required
github.com/pkg/errors@v0.7.0: not allowed by constraint ^0.8.0

# vendor is out of sync
github.com/pkg/errors: missing from vendor
github.com/aws-sdk-go/aws: hash of vendored tree not equal to digest in Gopkg.lock

dep check is also designed for use in automated tooling:

If any of its checks fail, it will exit 1. Passing -q will suppress any output, for maximum automated utility.
It's very fast; the checks it performs by default cannot hit the network. With a warm disk cache, it'll complete in seconds even on enormous projects.
cannot hit the network, which makes it very fast. Even a large project could use it as a git pre-commit hook:

You can use it as a git pre-commit hook, to keep you from committing an out-of-sync project. This will set it up:

cat >.git/hooks/pre-commit <<EOL
#!/bin/bash
dep check
EOL
chmod +x .git/hooks/pre-commit

It's also strongly recommended for use in CI. In dep itself, we replaced a hacky, slow and underinformative script with a single call to dep check.

`noverify`

Unfortunately, there are cases where you absolutely need to make modifications to certain projects in vendor, and getting the upstream project to change their ways just isn't practical. Code generation is probably the most common case.

In previous versions of dep, this was possible to do by wrapping dep ensure with a script that automatically re-applied your modifications afterwards. With vendor verification in place, though, dep will identify this as an aberrant state, dep ensure will always try to fix it, and dep check will always fail.

To address this, we have added noverify to Gopkg.toml, where you can provide a list of project roots (not packages) for which vendor verification should be skipped. Projects marked as such will not be rewritten for hash mismatches (though they still will if the solver picks a new version). dep check will still print a message about such issues so that you can still keep track of whether you actually are out of sync:

github.com/aws-sdk-go/aws: hash of vendored tree not equal to digest in Gopkg.lock (CHECK IGNORED: marked noverify in Gopkg.toml)

but if these "ignored" problems are the only ones dep check finds, it will exit 0.

dep, vgo/modules, and beyond

Modules, née vgo, which have been merged into the go command (behind experimental flags), and will be present in the release of Go1.11. The Go team believes this obviates the need for dep.

On the one hand, we're very glad that the Go team is finally taking dependency management problems seriously. And there are some profoundly useful ideas in vgo - significant contributions to the dependency management problem space, and ones that our future plans will certainly benefit from.

However, we believe that vgo pushes the line too far. In pursuit of algorithmic simplicity, it establishes rules that ask people to prioritize the ecosystem above their own goals, and push unnecessary work on already-stretched maintainers. These designs are so deeply baked into the toolchain that it will be impossible to use go without acquiescing to these rules.

That means there's no choosing between "vgo/modules or dep." It'll be "vgo, or another language."

This is a complicated topic. These writings look at the problems in depth, but are a lot to absorb. We are working to produce content that explain the problems in a more easily digestible way.

As we believe that the current incarnation of modules will be harmful to the Go community, we intend to continue with dep's development, moving towards an alternative prototype for the versioning behavior that currently undergirds the modules system. To that end, the primary focus in dep's next release will be changing the "get the newest version for transitive dependencies" problem. This issue is a cornerstone of the criticisms of dep; and it has been a goal of ours since before dep was first released.