plugin: Utility of Go plug-ins diminished by vendored dependencies

[akutz]

This issue is tied to the example project gpd.

cc @spf13

The Problem

The utility of Go plug-ins is almost completely erased by fact that many Go projects rely on vendored dependencies in order to ensure consistent build results.

The problem is pretty straight-forward. When an application (app) vendors a library (lib), the package path of the library is now path/to/app/vendor/path/to/lib. However, the plug-in is likely built against either path/to/lib or, if the plug-in vendors dependencies as well, path/to/plugin/vendor/path/to/lib.

This of course makes total sense and behaves exactly as one would expect with regards to Go packages. Despite the intent, these three packages are not the same:

• path/to/lib
• path/to/app/vendor/path/to/lib
• path/to/plugin/vendor/path/to/lib

While the behavior is consistent with regards to Go packages, it flies in the face of the utility provided by a combination of vendored dependencies and the new Go plug-in model.

Reproduction

This project makes it easy to reproduce the above issue.

Requirements

To reproduce this issue Go 1.8.x and a Linux host are required:

$ go version
go version go1.8.1 linux/amd64

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/akutz/go"
GORACE=""
GOROOT="/home/akutz/.go/1.8.1"
GOTOOLDIR="/home/akutz/.go/1.8.1/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build699913681=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

Download

On a Linux host use go get to fetch the gpd project:

$ go get github.com/akutz/gpd

Run the program

The root of the project is a Go command-line program. Running it will emit a message to the console:

$ go run main.go
Yes, we have no bananas,
We have no bananas today.

Build the plug-in

If the program is run with a single argument it is treated as the path to a Go plug-in. That plug-in is loaded and will emit a different message to the console. First, build the plug-in:

$ go build -buildmode plugin -o mod.so ./mod

To verify that the produced file is a plug-in, use the file command:

$ file mod.so
mod.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=8c78f9a393bd083bde91b2b34b8117592387f40e, not stripped

The file is reported as a shared object, verifying that it is indeed a Go plug-in.

Run the program with the plug-in

Run the program using the plug-in:

$ go run main.go mod.so
Yes there were thirty, thousand, pounds...
Of...bananas.

It works!

Vendor the shared dep package

However, what happens when the program vendors the shared dep package?

$ mkdir -p vendor/github.com/akutz/gpd && cp -r dep vendor/github.com/akutz/gpd
$ go run main.go mod.so
error: failed to load plugin: plugin.Open: plugin was built with a different version of package github.com/akutz/gpd/lib
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x504c45]

goroutine 1 [running]:
github.com/akutz/gpd/lib.NewModule(0x535498, 0x6, 0x539cb5, 0x21)
	/home/akutz/go/src/github.com/akutz/gpd/lib/lib.go:28 +0x55
main.main()
	/home/akutz/go/src/github.com/akutz/gpd/main.go:32 +0x13a
exit status 2

The program fails!

This is because the dep package includes a type that is used by both the shared lib package and the plug-in package, mod.

The plug-in linked against the lib package at github.com/akutz/gpd/lib which itself linked against the dep package at github.com/akutz/gpd/dep.

However, vendoring the dep package for the program causes the lib package as compiled into the program to link against github.com/akutz/gpd/vendor/github.com/akutz/gpd/dep, resulting in the program and the plug-in having two different versions of the lib package!

Vendor the shared lib package

However, what happens when the program vendors the shared lib package?

$ rm -fr vendor
$ mkdir -p vendor/github.com/akutz/gpd && cp -r lib vendor/github.com/akutz/gpd
$ go run main.go mod.so
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x504d65]

goroutine 1 [running]:
github.com/akutz/gpd/vendor/github.com/akutz/gpd/lib.NewModule(0x5355b8, 0x6, 0xc42000c2c0, 0x0)
	/home/akutz/go/src/github.com/akutz/gpd/vendor/github.com/akutz/gpd/lib/lib.go:28 +0x55
main.main()
	/home/akutz/go/src/github.com/akutz/gpd/main.go:32 +0x13a
exit status 2

The program fails! This is because the lib package contains a type registry that can be used to both register types and construct new instances of those types.

However, because the program's type registry is located in the package github.com/akutz/gpd/vendor/github.com/akutz/gpd/lib and the plug-in registered its type with github.com/akutz/gpd/lib, when the program requests a new object for the type mod_go, a nil exception occurs because the program and plug-in were accessing two different type registries!

The Hack

At the moment the only solution available is to create a build toolchain using a list of transitive dependencies generated from the application that is responsible for loading the plug-ins. This list of dependencies can be used to create a custom GOPATH against which any projects participating in the application must be built, including the application itself, any shared libraries, and the plug-ins.

The Solution

Is there one? Two possible solutions are:

1. Allow a src directory at the root of a vendor directory so that plug-ins can be built directly against a program's vendor directory. Today that would require a bind mount.
2. Allow plug-ins to link directly against the Go program binary that will load the programs.

Hopefully the Golang team can solve this issue as it really does prevent Go plug-ins from being useful in a world where applications are often required to vendor dependencies.

[akutz]

I also discovered this oddity: in addition to a plug-in needing to be built against the same sources, the value of GOPATH must also be the same:

# extract the GOPATH to two temp dirs
$ tar xzf ../rexray/gopath-0.9.0+15-1276481.tar.gz -C /tmp/tmp.9jaJCm8Aqd
$ tar xzf ../rexray/gopath-0.9.0+15-1276481.tar.gz -C /tmp/tmp.Na8C0iypRG

# no differences
$ diff -qr /tmp/tmp.9jaJCm8Aqd /tmp/tmp.Na8C0iypRG

# build plugin with tmp.9jaJCm8Aqd
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" go build \
  -tags "pflag gofig" -o /home/akutz/.libstorage/var/lib/mod/mock.so \
  -buildmode plugin ./mod

# build lsx-linux with tmp.Na8C0iypRG
$ GOPATH="/tmp/tmp.Na8C0iypRG:$GOPATH" ./gob ./cli/lsx/lsx-linux

# plug-in fails to load
$ ./lsx-linux
ERRO[0000] error opening module                          error=plugin.Open: plugin was built with a different version of package github.com/spf13/pflag path=/home/akutz/.libstorage/var/lib/mod/mock.so time=1495663533919

# build lsx-linux with tmp.9jaJCm8Aqd, same GOPATH used to build plug-in
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" ./gob ./cli/lsx/lsx-linux

# plug-in loads successfully
$ ./lsx-linux
usage: ./lsx-linux <executor> supported
                              instanceID
                              nextDevice
                              localDevices <scanType>
                              wait <scanType> <attachToken> <timeout>
                              mounts
                              mount [-l label] [-o options] device path
                              umount path

       executor:    mock
                    vfs

       scanType:    0,quick | 1,deep

       attachToken: <token>

       timeout:     30s | 1h | 5m

Even when using a common GOPATH to build the program and the plug-in, the contents of the GOPATH aren't enough, but they have to share the same root path as well. In essence, even if GOPATH=/tmp/1 and GOPATH=/tmp/2 are completely identical in what sources they have, down to the checksum of every file, if you build the program against one and the plug-in against the other, it fails. When the code is built, the value of GOPATH must be the same in both instances.

# remove the GOPATH used to build lsx-linux & mock.so
$ rm -fr /tmp/tmp.9jaJCm8Aqd

# recreate the GOPATH with identical contents
$ cp -fr /tmp/tmp.Na8C0iypRG /tmp/tmp.9jaJCm8Aqd

# rebuild lsx-linux
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" ./gob ./cli/lsx/lsx-linux

# the plug-in is loaded successfully
$ ./lsx-linux
usage: ./lsx-linux <executor> supported
                              instanceID
                              nextDevice
                              localDevices <scanType>
                              wait <scanType> <attachToken> <timeout>
                              mounts
                              mount [-l label] [-o options] device path
                              umount path

       executor:    mock
                    vfs

       scanType:    0,quick | 1,deep

       attachToken: <token>

       timeout:     30s | 1h | 5m

tl;dr: For plug-ins to work, not only must the sources in the GOPATH be the same for a plug-in and the program that loads it, the value of GOPATH must be the same when building the program and plug-in.

[akutz]

Hi,

It just occurred to me that the behavior outlined in the above comment results in the inability to build Go programs targeting a UNIX GOOS from a Windows host OS as the host can not have an identical GOPATH value as a UNIX host. Not unless using Cygwin or some other tool to mimic a UNIX path structure.

[jeremy-w]

Allow plug-ins to link directly against the Go program binary that will load the programs.

This sort of linking is directly supported by the Mach-O binary format used by Darwin (macOS/iOS/etc). The linker flag is -bundle_loader, and the ld(1) manpage documents it as:

      -bundle_loader executable
                 This specifies the executable that will be loading the bundle output file being linked.
                 Undefined symbols from the bundle are checked against the specified executable like it was
                 one of the dynamic libraries the bundle was linked with.

This is used when building a special "dynamically bound bundle" file type, which is a separate file type from any of a shared library, static library, or executable.

I guess that bodes well for when plugin support gets added for Darwin, but I don't know that ELF has baked in support for plugins using their host binary's names like that.

[akutz]

Hi @jeremy-w,

Interesting. That is very useful...for Darwin. Which doesn't yet have Go plug-in support :) Which sucks, because Darwin is my primary development platform...

For now, this is what I'm doing:

• Declaring that all participants in my software ecosystem (projects/products) must define their GOPATH as /tmp/go if they wish to either load or be plug-ins thanks to the issue I discovered in this comment above.
• On Travis-CI I'm using the following steps to copy my vendored sources to the appropriate location:

  - glide install
  - export PROJECT_NAME="rexray"
  - export GOPATH_OLD="$GOPATH"
  - export GOPATH="/tmp/go"
  - mkdir -p "$GOPATH"/{bin,pkg,src}
  - mv "$GOPATH_OLD"/bin/* "$GOPATH"/bin/
  - export PATH="${GOPATH}/bin:${PATH}"
  - mkdir -p "$GOPATH"/src/github.com/codedellemc
  - rsync -ax vendor/ "$GOPATH"/src/ && rm -fr vendor
  - cd .. && mv "$PROJECT_NAME" "$GOPATH"/src/github.com/codedellemc/
  - cd "$GOPATH"/src/github.com/codedellemc/"$PROJECT_NAME"

• Before I remove the vendor directory I use it to create a tarball to make it easy for others to build plug-ins. The below commands use tar's transform functionality to change the leading path from vendor/ to src/, making it easy to treat the contents of the tarball, once extracted, as a GOPATH:

Darwin

tar \
  --exclude '.git' \
  --exclude 'vendor' \
  -czf "gopath.tar.gz" \
  -s ',^./,src/,' \
  -C vendor .

Linux

tar \
  --exclude '.git' \
  --exclude 'vendor' \
  -czf "gopath.tar.gz" \
  --xform 's,^./,src/,S' \
  -C vendor .

[mattfarina]

@akutz What does a dependency update strategy look like for an application and its plugins?

[akutz]

Hi @mattfarina,

I'm moving ahead with a different approach that renders your question moot. You'll see why in a minute.

[akutz]

Hi @spf13 / @mattfarina / @bradfitz / @jeremy-w,

As I just indicated to @mattfarina, I am moving forward with a solution that essentially circumvents the original problem. The complete details can be found at akutz/gpds. For convenience I am copying the document below. For the members of the Go team, I am really curious about the section Curious Exception. It's something I never considered, and if that's not weird enough, check out Exception to the Exception.

---

Go Plug-ins & Vendored Dependencies: A Solution

This document outlines a solution for the problem described in golang/go#20481. Please review the original problem description before continuing.

The problem was fairly straight-forward and ultimately so is the solution: a Go plug-in should not depend on a host process's symbols. That means:

• Go plug-ins should use a unidirectional model for type registration
• Go plug-ins should use interface{} for all non-stdlib types involved
  in ingress and egress host-plug-in communications

Unidirectional Model

Go supports the dependency inversion principle (DIP) through the use of interface abstractions, but there still must exist a mechanism to provide implementations of the abstractions on which a program depends. One such solution can be found in the list of suggested implementations of inversion of control (IoC): the service locator pattern.

The service locator pattern is very easy to implement in Go as a simple type registry. Consumers that require an implementation of some interface are able to query the type registry and receive an object instance that fulfills the abstraction. There are two models that can be used to prime the registry with types: bidirectional and unidirectional.

The above diagram is an example of the bidirectional model, but it fails when used in concert with Go plug-ins due to the issues with dependencies outlined in golang/go#20481. The solution is a unidirectional model:

Illustrated in the diagram above, the unidirectional model provides the same type registry that the bidirectional model does but relocates type registration from the plug-ins' init functions to the host process. This change means the plug-ins no longer depend on the type registry in the host process, and that's very important because a plug-in cannot depend on a host process's symbols.

Interface In / Interface Out

Go interfaces are really powerful, but they are also quick to cause issues when used with plug-ins for two reasons:

1. Interface equality is not as simple as it seems
2. The fully-qualified path to an interface matters

Interface Equality

The following examples demonstrate the power and peril of using Go interfaces interchangeably having assumed equality. The first example defines two, identical interfaces, dog and fox, and two structs, best friends that implement the interfaces, copper and todd (run example):

package main

import (
	"fmt"
)

type dog interface {
	bark() string
}

type fox interface {
	bark() string
}

type copper struct{}

func (c *copper) bark() string { return "woof!" }

type todd struct{}

func (t *todd) bark() string { return "woof!" }

func barkWithDog(d dog) { fmt.Println(d.bark()) }
func barkWithFox(f fox) { fmt.Println(f.bark()) }

func main() {
	var d dog = &copper{}
	var f fox = &todd{}
	barkWithDog(d)
	barkWithFox(f)
}

The above code, when executed, will print woof! on two lines. The first line is the result of the dog Copper barking, and the second line is his friend Todd the fox taking a turn. However, what makes Copper a dog or Todd a fox? According to the code it's because copper implements the function bark() string from the dog interface and todd implements the same function from the fox interface.

Does that mean that copper and todd are interchangeable? In fact, the two friends decided to pretend to be one another in order to play a trick on the kind old lady and hunter (run example):

func main() {
	var d dog = &todd{}
	var f fox = &copper{}
	barkWithDog(f)
	barkWithFox(d)
}

How can Todd be a fox and Copper a dog? According to Go's interface rules, a variable of type fox can be assigned any type that implements the bark() string function. A function that has an
argument of type dog or fox can also accept any type that implements the bark() string function, even if that type is another interface.

It would appear then that multiple Go interfaces, if they define the same abstraction, are identical. However, thanks to Go's strong type system, interfaces are not as interchangeable as they first appear (run example):

package main

import (
	"fmt"
)

type dog interface {
	bark() string
	same(d dog) bool
}

type fox interface {
	bark() string
	same(f fox) bool
}

type copper struct{}

func (c *copper) bark() string    { return "woof!" }
func (c *copper) same(d dog) bool { return c == d }

type todd struct{}

func (t *todd) bark() string    { return "woof!" }
func (t *todd) same(f fox) bool { return t == f }

func barkWithDog(d dog) { fmt.Println(d.bark()) }
func barkWithFox(f fox) { fmt.Println(f.bark()) }

func main() {
	var d dog = &todd{}
	var f fox = &copper{}
	barkWithDog(f)
	barkWithFox(d)
}

The above example will no longer emit the sound of two friends barking, but rather the following errors:

tmp/sandbox006620983/main.go:31: cannot use todd literal (type *todd) as type dog in assignment:
	*todd does not implement dog (wrong type for same method)
		have same(fox) bool
		want same(dog) bool
tmp/sandbox006620983/main.go:32: cannot use copper literal (type *copper) as type fox in assignment:
	*copper does not implement fox (wrong type for same method)
		have same(dog) bool
		want same(fox) bool
tmp/sandbox006620983/main.go:33: cannot use f (type fox) as type dog in argument to barkWithDog:
	fox does not implement dog (wrong type for same method)
		have same(fox) bool
		want same(dog) bool
tmp/sandbox006620983/main.go:34: cannot use d (type dog) as type fox in argument to barkWithFox:
	dog does not implement fox (wrong type for same method)
		have same(dog) bool
		want same(fox) bool

The relevant piece of information from the above error text is the following:

have same(fox) bool
want same(dog) bool

In other words, even though Go interfaces A and B are identical, A{A} and B{B} are not. If A==B and C==D, A{C} != B{D}.

Because of this rule, without a shared types library, even with Go interfaces, it's not possible for Go plug-ins to expect to share or use symbols provided by the host process.

Fully-Qualified Type Path

However, even redefining interfaces inside plug-ins to match types found in the host process will fail if those interfaces are used by exported symbols. This section uses this project's dog package. The following command will get the package and build its plug-ins:

$ go get github.com/akutz/gpds && \
  cd $GOPATH/src/github.com/akutz/gpds/dog && \
  for d in $(find . -maxdepth 1 -type d | grep -v '^.$'); do \
    go build -buildmode plugin -o $d.so $d; \
  done

Run the program using the sit.so plug-in:

$ go run main.go dog.go sit.so
error: invalid Command func: func(main.dog)
exit status 1

An error occurs because the sit.so plug-in defines an interface dog to match the host program's interface Dog. Both interfaces include a single function: Name() string. However, these types are different because their fully-qualified type paths (FQTP) are not the same. An FQTP includes the package path to a type and the type's name, where the name is case sensitive (since case sensitivity is used by Go to indicate public and private members).

Therefore invoking the Command(Dog) function fails, because while the interface definitions are identical with regards to the equality ruleset outlined above, the two interfaces do not have the same FQTP.

Curious Exception

There is one curious exception to this rule: when an interface is defined in the main package of the host program as well as the main package of the plug-in. Run the program using the speak.so plug-in:

$ go run main.go dog.go speak.so
Lucy

The program should have printed the name "Lucy". However, if the code is examined, the Dog interface is defined in both the host program and in the plug-in package. Yet it works. Why? The answer is almost so embarrassingly obvious that it makes this author hesitant to admit it took him an hour of looking at the problem to figure it out.

Both interfaces have a fully-qualified package path of main.Dog.

When interfaces are defined in the main package of the hosting program and in the main package of a plug-in, their symbols are identical. However, like most things, there's an exception to even this.

Exception to the Exception

What happens if the Dog interface references itself? The answer is an error this author has never seen before in his history of working with the Go programming language. To reproduce this error, run the program using the stay.so plug-in:

$ go run main.go self.go stay.so
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x534aad, 0xe)
	/home/akutz/.go/1.8.1/src/runtime/panic.go:596 +0x95
runtime.newstack(0x0)
	/home/akutz/.go/1.8.1/src/runtime/stack.go:1089 +0x3f2
runtime.morestack()
	/home/akutz/.go/1.8.1/src/runtime/asm_amd64.s:398 +0x86

goroutine 1 [running]:
runtime.(*_type).string(0x7f2488279520, 0x0, 0x0)
	/home/akutz/.go/1.8.1/src/runtime/type.go:45 +0xad fp=0xc44009c358 sp=0xc44009c350
runtime.typesEqual(0x7f2488279520, 0x51d0c0, 0x50a310)
	/home/akutz/.go/1.8.1/src/runtime/type.go:543 +0x73 fp=0xc44009c480 sp=0xc44009c358
runtime.typesEqual(0x7f2488270740, 0x5137c0, 0x5137c0)
	/home/akutz/.go/1.8.1/src/runtime/type.go:586 +0x368 fp=0xc44009c5a8 sp=0xc44009c480
runtime.typesEqual(0x7f2488279520, 0x51d0c0, 0x50a310)
	/home/akutz/.go/1.8.1/src/runtime/type.go:615 +0x740 fp=0xc44009c6d0 sp=0xc44009c5a8
...additional frames elided...

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/home/akutz/.go/1.8.1/src/runtime/asm_amd64.s:2197 +0x1
exit status 2

The above program fails due to a Go runtime panic where Go is recursively trying to determine if the main.Dog interface from the host program is the same type as the main.Dog interface defined in the plug-in. The interfaces were considered the same when they did not reference themselves with their respective Self() Dog functions.

The Solution

The proposed solution adheres to the crucial restriction outlined at the beginning of this document -- Go plug-ins should not depend on a host program's symbols. This project is used to demonstrate a program and plug-ins that:

• Use a unidirectional model for type registration
• Use interface{} for all non-stdlib types involved in ingress and egress
  host-plug-in communications

To get started please clone this repository:

$ go get github.com/akutz/gpds && cd $GOPATH/src/github.com/akutz/gpds

Running the program will emit a little ditty by Mr. Chapin:

$ go run main.go
Yes, we have no bananas,
We have no bananas today.

Next, build the plug-in mod.so:

$ go build -buildmode plugin -o mod.so ./mod

Running the program with the plug-in will cause the output to be somewhat altered:

$ go run main.go mod.so
*main.v2Config
Yes there were thirty, thousand, pounds...
Of...bananas.

*main.v2Config
Bottom-line, sh*t kicking country choir
You'll see your part come by

The above steps do not appear to illustrate anything too fancy, but under the covers is a model that enables Go plug-ins to work alongside vendoered dependencies with ease. Pulling back the covers ever so slightly reveals how it all works.

Lib

For starters, this is the Module interface defined in ./lib/lib.go:

// Module is the interface implemented by types that
// register themselves as modular plug-ins.
type Module interface {

	// Init initializes the module.
	//
	// The config argument can be asserted as an implementation of the
	// of the github.com/akutz/gpds/lib/v2.Config interface or older.
	Init(ctx context.Context, config interface{}) error
}

A Module interface includes a single function, Init, which accepts a Go context and second argument of type interface{}. The second argument is expected to be a sort of configuration provider, provided to plug-ins to inform their initialization routine. Please note the Godoc for the argument:

The config argument can be asserted as an implementation of the
of the github.com/akutz/gpd/lib/v2.Config interface or older.

The documentation indicates which type the argument can be asserted as, and more importantly explains that the object provided can be asserted as a specific version of that type or older. All types should be versioned and plug-ins that assert v1.Type should continue to work even if v1+.Type is provided.

Please note that this model could be further enhanced so that plug-ins provide a symbol that contains the expected API version so that host programs can eventually deprecate older types by restricting which plug-ins get loaded based on their supported API type.

Mod

The file ./mod/mod.go is the core of the plug-in. At the top of the
file is the Types symbol:

// Types is the symbol the host process uses to
// retrieve the plug-in's type map
var Types = map[string]func() interface{}{
	"mod_go": func() interface{} { return &module{} },
}

The Types symbol is very important, and the model proposed in this project expects all plug-ins to define this symbol. The symbol is a type map that is used by the host program to register the plug-in's type names and functions to construct them.

The module mod_go referenced in the plug-in's type map looks like this:

type module struct{}

func (m *module) Init(ctx context.Context, configObj interface{}) error {
	config, configOk := configObj.(Config)
	if !configOk {
		return errInvalidConfig
	}
	fmt.Fprintf(os.Stdout, "%T\n", config)
	fmt.Fprintln(os.Stdout, config.Get(ctx, "bananas"))
	return nil
}

Since this is an example the plug-in's module only defines a barebones implementation of the Module interface. The module's Init function first asserts that the provided configObj argument can be asserted as the Config interface and then uses the typed object to retrieve and print bananas.

But wait, how is the plug-in able to assert the interface Config if the plug-in is not sharing any symbols with the host process? Simple, the plug-in simply treats the sources in the versioned lib package as C headers and copies the v1 or v2 headers into the plug-in's own package.

Conclusion

Hopefully this document has not only shown how to solve the issue of Go plug-ins and vendored dependencies, but also clearly articulated the reasoning behind the decisions that led to the proposed solution.

[dlsniper]

I believe this is a duplicate of #18827, no?