proposal: cmd/go: automatic and partial vendoring in module mode

[bcmills]

This proposal overlaps with (and hopefully unifies) several existing issues, linked in the text below.

I'd like to implement it soon, in the 1.13 1.14 cycle, so if you have feedback please do respond quickly. 🙂

Problem summary

Users want a durable, local view of their source code that works with existing diff tools and does not require per-user configuration in cloned repositories.

• Relying on module proxies does not necessarily satisfy delivery contracts.
• Saved module caches do not interoperate well with version-control and code-review tools.
• -mod=vendor requires configuration per user (GOFLAGS) or per invocation, and makes it too easy to ship code that produces a different build in vendored mode than in the normal module mode.

Proposal

Under this proposal, the source code for the packages listed in vendor/modules.txt — and the go.mod files for the modules listed in vendor/modules.txt, if any — will be drawn from the vendor directory automatically (#27227).

If a replace directive in the main module specifies a module path, the module source code will be vendored under the path that provides the replacement, not the path being replaced. That preserves the 1:1 correspondence between import paths and filesystem directories, while allowing replacement targets to alias other modules (#26904). If a replace directive specifies a file path, then either that path must be outside the vendor directory or the vendor/modules.txt file must not exist (#29169).

Package patterns such as all and example.com/... will match only the packages that are present in the vendor directory, not unvendored packages from the same module. During the build, if additional packages from the vendored modules are needed in order to satisfy an import, the source for those packages will be fetched (from the module cache, if available) and added to the vendor directory. (Packages from outside the already-vendored modules will not be vendored automatically.)

Any time the go.mod file is written, if a module path found in vendor/modules.txt has a different version than that found in the build list, the already-vendored packages and go.mod file from the previous version will be deleted, and updated versions of those packages will be written in their place (#29058). Transitive imports of those packages will be resolved, and may populate additional packages in other already-vendored modules.

If go get removes a module from the build list entirely, its package source and go.mod file will be removed, but an entry for the module (with version none) will remain in vendor/modules.txt. That way, if a future operation (such as a go get or go build) adds the module to the build list again, it will remain vendored as before.

When go mod tidy is run, it will add or remove packages from the vendor directory so that it continues to contain only the subset of packages found in the transitive import graph. It will also remove go.mod files and entries in vendor/modules.txt for modules that are no longer present in the build list.

To encourage the minimal use of vendor directories, the go mod vendor subcommand will accept an optional list of packages or modules. go mod vendor <module> will update the vendor directory to contain the go.mod file for <module> and source code for its packages that appear in the transitive import graph of the main module. (Note that, since the criterion for inclusion of a package is its existence in the import graph, vendoring in an additional module should not affect the contents of any previously-vendored modules.)

go mod vendor <pattern> for an arbitrary module pattern will add # <pattern> to vendor/modules.txt, and vendor in the go.mod files (and any packages found in the import graph) for modules matching <pattern>, adding individual comments to vendor/modules.txt for those modules.

Note in particular that go mod vendor all will copy in go.mod files for all of the module dependencies in the module graph (and add entries in vendor/modules.txt for those modules). That ensures that after go mod vendor all, go list can produce accurate results without making any further network requests (see also #19234 and #29772).

The go mod vendor subcommand will accept a new flag, -d. go mod vendor -d <pattern> will remove all previously-vendored modules matching <pattern> from the vendor directory (and from vendor/modules.txt), as well as any previously-stored patterns matching those modules (including <pattern> itself, if present).

go mod vendor, without further arguments, is equivalent to go mod vendor all. go mod vendor -d is equivalent to go mod vendor -d all. If go mod vendor -d causes vendor/modules.txt to become empty, it will also remove the entire vendor directory.

---

Edits

• Made module patterns sticky per proposal: cmd/go: automatic and partial vendoring in module mode #30240 (comment).

[bcmills]

(CC @jayconrod @rasky @JeremyLoy @theckman)

[FiloSottile]

To encourage the minimal use of vendor directories, ...

Why are partial vendor folders something we want to encourage? Most use cases listed here and in the linked issues would require all dependencies vendored at all times.

Also, can you clarify if go get or go mod tidy would ever add new modules to the vendor folder, or if running go mod vendor would still be required after every new dependency is added in order to avoid a partial vendor folder?

[bcmills]

Why are partial vendor folders something we want to encourage? Most use cases listed here and in the linked issues would require all dependencies vendored at all times.

Some dependencies are more robust than others. For example, you might trust github.com to be generally available, but want to vendor in dependencies that happen to be hosted using bzr or svn so that you don't have to install those tools on every machine that will build your module.

[bcmills]

Also, can you clarify if go get or go mod tidy would ever add new modules to the vendor folder, or if running go mod vendor would still be required after every new dependency is added in order to avoid a partial vendor folder?

go get and go mod tidy would not add dependencies to vendor automatically.

We could perhaps make go mod vendor (without arguments) set some flag in modules.txt to indicate that all additional modules should be vendored.

[bcmills]

More generally, though, the main goal of automatic vendor updates is to prevent version skew. Copying in newly-added modules does not further that goal, since there are no out-of-date contents in the first place.

[thepudds]

I would suspect the most common use case might be vendoring 100% of dependencies?

If so, and if vendoring in 1.13 is going to be able to track updates via go get and go mod tidy in some cases, it would seem that once you have signaled you want automated tracking it likely should be the default behavior at that point to be 100% complete in any automated tracking, rather than defaulting to partial tracking? (For example, track all updates after a go mod vendor with no args, as you suggested two comments back)?

[FiloSottile]

It definitely makes sense to support partial ones. I just suspect (and might be wrong!) that 90% of users opting into vendoring really mean to vendor everything, and a reasonable chunk of that 90% would be surprised by it behaving otherwise.

[ianthehat]

In the presence of a reliable proxy, I can't think of any reasonable cases for a partial vendor directory, and lots of possible confusion.
I would personally argue we go in the other direction, as in if you try to build in vendor mode it is not allowed to see anything outside the current module (except the stdlib)

[bcmills]

@ianthehat, one use-cases for vendoring, given proxies, is to vendor in private code for which the proxies do not have access.

For example, a contract-based startup might want to vendor in their proprietary utility modules before delivering the code to their customers.

[thepudds]

@bcmills Could you comment on the interplay with -mod=readonly, and/or options to disable automatic downloads for people who would prefer to fail if vendor is missing something?