v0.5.0 of dep has been released!
The big theme of this release is performance improvements. dep was designed for safety from the outset, because we knew that foundation would let us speed things up later. Now we have!
NOTE: your whole team will need to update at once to this new release, as it results in changes to the structure of
Gopkg.lock that older versions of dep won't know how to work with.
There are two big aspects to the performance improvements: source metadata caching, and vendor verification.
Source metadata caching is an experimental feature that caches the result of all the parsing and code-backed analysis dep does as part of the solving process: reading in your dependencies'
Gopkg.toml files, parsing the .go files for
import statements, etc. All that work, and the
git checkout necessary to put code on disk to analyze, is what made the solver plod along in the past.
With the caching enabled (managed by the env var
DEPCACHEAGE), any combination of version and project that was already visited is retrieved from a persistent cache. Time per solving step drops to the (sub-)millisecond range; previously it was on the order of hundreds of milliseconds or seconds.
Vendor verification is the notion that
Gopkg.lock should contain enough information to be able to verify whether the current contents of
vendor/ are exactly as they should be, including whatever pruning options you've set. We've now done this, by adding the
pruneopts fields to each
[[project]] stanza in
The performance impact of all this is that it is no longer necessary for dep to rewrite the entirety of
vendor/ on every
dep ensure run. Instead, dep selectively writes out or removes only the files necessary to bring
vendor/ back in line with
-v, it'll also tell you why change was made:
# Bringing vendor into sync (1/4) Wrote firstname.lastname@example.org: version changed (was v1.0.0) (2/4) Wrote github.com/gregjones/httpcache@master: revision changed (2bcd89a174 -> 9cad4c3443) (3/4) Wrote github.com/prometheus/common@master: prune options changed (UT -> NUT) (4/4) Removed unused project github.com/kr/pretty
While the improvements affect different workflows in different ways, a representative
dep ensure -v run (including both a solve and updating
vendor/) for CockroachDB dropped from 120s to 4s in local benchmarking.
Vendor verification has implications beyond just performance. With it complete, we fixed dep's final blind spot on whether all of the dependency-relevant information in your project -
imports in code,
vendor/ - are in sync. That enables not only the granular feedback about
vendor/ changes above, but it also lets us tell you exactly what changed in your project that pushed it out of sync, causing a solve.
dep informed you of this in the past, but it was kinda useless:
$ dep ensure -update -v Warning: Gopkg.lock is out of sync with Gopkg.toml or the project's imports.
Not very helpful.
Now, though, if
dep ensure -v sees your project is out of sync in a way that entails re-solving the graph, it will tell you exactly why:
$ dep ensure -v # Gopkg.lock is out of sync github.com/kr/pretty: imported or required, but missing from Gopkg.lock's input-imports github.com/aws-sdk-go/aws/awserr: in Gopkg.lock's input-imports, but neither imported nor required email@example.com: not allowed by constraint ^0.8.0
Of course, what if you just want to know what's out of sync, without actually changing anything? We have a new subcommand for that!
This release introduces a new subcommand,
dep check, which reports all the ways that your project is out of sync. This includes the output of
dep ensure -v, but also looks for any issues in
$ dep check # Gopkg.lock is out of sync github.com/kr/pretty: imported or required, but missing from Gopkg.lock's input-imports github.com/aws-sdk-go/aws/awserr: in Gopkg.lock's input-imports, but neither imported nor required firstname.lastname@example.org: not allowed by constraint ^0.8.0 # vendor is out of sync github.com/pkg/errors: missing from vendor github.com/aws-sdk-go/aws: hash of vendored tree not equal to digest in Gopkg.lock
dep check is also designed for use in automated tooling:
- If any of its checks fail, it will exit 1. Passing
-qwill suppress any output, for maximum automated utility.
- It's very fast; the checks it performs by default cannot hit the network. With a warm disk cache, it'll complete in seconds even on enormous projects.
- cannot hit the network, which makes it very fast. Even a large project could use it as a git pre-commit hook:
You can use it as a git pre-commit hook, to keep you from committing an out-of-sync project. This will set it up:
cat >.git/hooks/pre-commit <<EOLdep check EOL chmod +x .git/hooks/pre-commit
It's also strongly recommended for use in CI. In dep itself, we replaced a hacky, slow and underinformative script with a single call to
Unfortunately, there are cases where you absolutely need to make modifications to certain projects in vendor, and getting the upstream project to change their ways just isn't practical. Code generation is probably the most common case.
In previous versions of dep, this was possible to do by wrapping
dep ensure with a script that automatically re-applied your modifications afterwards. With vendor verification in place, though, dep will identify this as an aberrant state,
dep ensure will always try to fix it, and
dep check will always fail.
To address this, we have added
Gopkg.toml, where you can provide a list of project roots (not packages) for which vendor verification should be skipped. Projects marked as such will not be rewritten for hash mismatches (though they still will if the solver picks a new version).
dep check will still print a message about such issues so that you can still keep track of whether you actually are out of sync:
github.com/aws-sdk-go/aws: hash of vendored tree not equal to digest in Gopkg.lock (CHECK IGNORED: marked noverify in Gopkg.toml)
but if these "ignored" problems are the only ones
dep check finds, it will exit 0.
dep, vgo/modules, and beyond
Modules, née vgo, which have been merged into the
go command (behind experimental flags), and will be present in the release of Go1.11. The Go team believes this obviates the need for dep.
On the one hand, we're very glad that the Go team is finally taking dependency management problems seriously. And there are some profoundly useful ideas in vgo - significant contributions to the dependency management problem space, and ones that our future plans will certainly benefit from.
However, we believe that vgo pushes the line too far. In pursuit of algorithmic simplicity, it establishes rules that ask people to prioritize the ecosystem above their own goals, and push unnecessary work on already-stretched maintainers. These designs are so deeply baked into the toolchain that it will be impossible to use
go without acquiescing to these rules.
That means there's no choosing between "vgo/modules or dep." It'll be "vgo, or another language."
This is a complicated topic. These writings look at the problems in depth, but are a lot to absorb. We are working to produce content that explain the problems in a more easily digestible way.
As we believe that the current incarnation of modules will be harmful to the Go community, we intend to continue with dep's development, moving towards an alternative prototype for the versioning behavior that currently undergirds the modules system. To that end, the primary focus in dep's next release will be changing the "get the newest version for transitive dependencies" problem. This issue is a cornerstone of the criticisms of dep; and it has been a goal of ours since before dep was first released.