Unable to debug/troubleshooting tests of projects built with v3/go-alpha plugin

[camilamacedo86]

Bug Report

What did you do?

• Installed Kubuduilder without installing the binaries since it should no longer require any more for v3/:

os=$(go env GOOS)
arch=$(go env GOARCH)

# download kubebuilder and extract it to tmp
curl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/

# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

• Follow up Quick Start with the plugin v3/go-alpha
• Then, execute the test suite_test.go via the IDE to debug it
  OR
• Try to execute the tests without the Makefile target go test ./...

What did you expect to see?

• Able to execute the test via the IDE as it worked before for v2 plugin
• Able to execute the tests directly via go test ./... as it worked before for v2 plugin

What did you see instead? Under which circumstances?

An error that is NOT intuitive and meaningful because EnvTest was not able to find the binaries:

Possible Solution

        By("setting binaries")
	if os.Getenv("KUBEBUILDER_ASSETS") == "" {
		binaryAssetsPath, err := filepath.Abs(filepath.Join("..", "..", "bin"))
		Expect(err).NotTo(HaveOccurred())
		testEnv.BinaryAssetsDirectory = binaryAssetsPath
	}

PR: #1893

Would the above solve all scenarios? No.

See that it still with a pitiful scenario. If we do not execute the first make test locally will NOT have the bins locally and we still unable to execute the tests without using the makefile target.

is fine we ask for the users run make tests before try to debug on the IDE?

it shows ok as a workaround. However, it shows not to be good enough as the ultimate solution.

Note that it is NOT intuitive for the users. As I user of Kubebuilder, I expected that the tool provide a much more readable, intuitive and understandable code to reduce the learning curve and keep its maintainability.

Then, I'd prefer we have an specific target to setup the binaries and replace:

# Run tests
ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
test: generate fmt vet manifests
	mkdir -p ${ENVTEST_ASSETS_DIR}
	test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/{{ .ControllerRuntimeVersion }}/hack/setup-envtest.sh
	source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

With:

# Run tests
test: generate fmt vet manifests tooling 
	go test ./... -coverprofile cover.out

# Setup the local envinroment 
tooling:
	cd hack/; go run tooling.go 

The solution proposed on #1711

Additional context

This BUG was introduced with the change to download the binaries via shell script from controller-runtime. See: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v3/Makefile#L16-L22

This happens because by default EnvTest will look for the binaries in /usr/local/kubebuilder/bin/ and the test ONLY works via the target because we are source setup-envtest.sh.

[estroz]

Relates to kubernetes-sigs/controller-runtime#1234

[camilamacedo86]

Closing it a favour to kubernetes-sigs/controller-runtime#1234 and kubernetes-sigs/controller-runtime#1300 as discussed in the bug triage meeting.