cmd/go: 'go run' should run executables in module mode outside a module

[eliasnaur]

#40276 implements go install path@version for installing a Go binary outside a module. I propose the same support added to go run, with equivalent behavior. That is,

$ go run gioui.org/cmd/gogio@d5bdf0756a5a

should build and run the gioui.org/cmd/gogio program at version d5bdf0756a5a

Why isn't go install enough for my uses? Consider a README describing how to build and use an auxiliary Go program:

To build the XYZ Android app you need to use the gogio tool:

$ export PATH=$PATH:$GOPATH/bin
$ go install gioui.org/cmd/gogio@d5bdf0756a5a
$ gogio -target android example.com/cmd/xyz

The README has several issues:

1. go install'ing the binary is reproducible, but running it isn't. For example, the user may have an old gogio in their PATH already, and fail to run the instructed go install. They may remember, but later install a different version of gogio.
2. go install polutes the user's GOPATH/bin, and PATH if it includes GOPATH/bin.
3. If GOPATH is not set, the README has to contain hardcoded paths (~/go/bin).

In contrast, with go run path@version support, the README is reduced to just:

To build the XYZ Android app you need to use the gogio tool:

$ go run gioui.org/cmd/gogio@d5bdf0756a5a -target android example.com/cmd/xyz

[myitcv]

cc @bcmills @jayconrod @ianthehat given previous discussion on this

[bcmills]

Just to consider as an alternative:

example.com$ GOBIN=$(pwd) go install gioui.org/cmd/gogio@d5bdf0756a5a

example.com$ ./gogio
gogio: specify a package

That has its own problems (namely, $(pwd) is not portable), but it makes clear that for subsequent invocations the user should re-invoke the compiled binary rather than re-resolving and recompiling from upstream.

If we allowed go build to accept the @version syntax, then it could be made portable:

$ go build -o ./gogio.exe gioui.org/cmd/gogio@d5bdf0756a5a
$ ./gogio.exe

[ianthehat]

The go install thing is not portable, because of the varying executable filename, and I think adding this support to go build would be a mistake, I think would rather see us add a -o to go install if that is where we are going.
One of the use cases I find more interesting to talk about is writing reproducible generate lines

  //go:generate go run golang.org/x/tools/cmd/stringer@v1.2.3 -type=Pill

[eliasnaur]

The go install thing is not portable, because of the varying executable filename, and I think adding this support to go build would be a mistake, I think would rather see us add a -o to go install if that is where we are going.
One of the use cases I find more interesting to talk about is writing reproducible generate lines

  //go:generate go run golang.org/x/tools/cmd/stringer@v1.2.3 -type=Pill

What if you have several such invocations of the same tool, each with a duplicate @version modifier?

I omitted the go:generate use-case from this proposal because I think it's better to have such dependencies recorded in the go.mod file by using the idiom of _-importing the tool in a tools.go file.

[ianthehat]

Why would duplicate @version modifiers be a problem?

The _ import is bad because it causes the tool to modify the version selection of your main application, and also to pull things into your dependency graph that your binary does not actually depend on. It also causes the tools to affect each other, rather than being run with the versions the author has tested with. In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

[jayconrod]

cc @matloob as well.

Personally I'm in favor of go run supporting this with the same semantics and restrictions as #40276. This came up a few times in the discussion of #40276. Let's set aside go build.

The only technical barrier is that we'd need to cache linked binaries with a different eviction policy than compiled packages. Currently, we don't cache linked binaries at all.

Other than that, it's just a question of CLI design and impact to the ecosystem.

[eliasnaur]

Why would duplicate @version modifiers be a problem?

I was referring to having to update all versions if you want a newer version of the tool. Maybe that's not too bad.

The _ import is bad because it causes the tool to modify the version selection of your main application, and also to pull things into your dependency graph that your binary does not actually depend on. It also causes the tools to affect each other, rather than being run with the versions the author has tested with. In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

Good points. In an ideal world, go:generate dependencies should be recorded in go.mod, but the downsides you point out seem to outweigh the advantages.

[ianthehat]

I think if there was a reasonably common pattern of //go:generate go run package@version args... we could easily write tooling to maintain those lines separately if it turns out to be needed.

[myitcv]

In general it is an acceptable hack while we don't have a better answer, but not a long term acceptable solution in my opinion.

The argument about dependencies being varied to versions not tested by the author might equally apply to any third party library you are using, so that doesn't sway the argument for me. The fact that, under such a scheme, we would have multiple sites at which to maintain tool versions is a real problem however. Because use of these tools is by no means limited to go:generate directives, e.g. scripts.

[ianthehat]

There is a significant difference between a library you want to include in your code that shares dependancies with other libraries in your graph, and a binary you want to run exactly as the author intended it to be run. The module story has been very focused on the former (for good reason) and the existing approaches have not left people happy with the results for the latter, which is one of the reasons we have talked about these kinds of changes.

I am mostly uninterested in scripts or makefiles because I think they already have all the tools they need, the path and install hacks are good enough for those cases, it might not be beautiful and need some extra lines, but I don't find that a big deal.

[peebs]

The argument about dependencies being varied to versions not tested by the author might equally apply to any third party library you are using, so that doesn't sway the argument for me. The fact that, under such a scheme, we would have multiple sites at which to maintain tool versions is a real problem however. Because use of these tools is by no means limited to go:generate directives, e.g. scripts.

I also see a significant difference between depending on libraries and installing a released and versioned binary.

Depending on a library means to me taking ownership of how the library fits into your dep graph and sufficiently testing your code to be confident in the potentially unique dep graph.

I never want to modify a released binary's deps based on independent local code i'm developing. If I need to modify the deps of a released binary i'm either in the process of forking or contributing upsteam to the project directly. I want control over the deps of code I am currently authoring only. If I am building a main package outside of my current project then I want a universally reproducible artifact as much as possible.