✨ EnvTest Binaries Setup Tool #1488

DirectXMan12 · 2021-04-21T07:15:09Z

This introduces a tool (in Go) to manage installed versions of the envtest binaries.
It can download new ones (both always and if necessary), enumerate installed and available versions, and remove versions.

By default, it installs to an OS-specific cache dir (as per OS conventions), but that can be customized if need-be.

The tool itself is in a separate module, so it can just be go installed-ed on 1.16 w/o pulling in extra deps.

The README in the subdirectory and the command's --help flag have more details, but the TL;DR of usage is:

go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
source <(setup-envtest fetch -k '1.19.*' -p env)
# or, for the latest
source <(setup-envtest fetch -p env)

Fixes #1300

DirectXMan12 · 2021-04-21T07:15:37Z

/assign @joelanford
/assign @camilamacedo86

DirectXMan12 · 2021-04-21T07:36:49Z

Quick FAQ on downloading and "latest":

fetch is lazy when given a non-latest version (e.g. -k '1.19.*' or -k 1.19.2) -- if you already have something on disk that matches, it won't make any network connections
fetch can be told to fetch the latest in general (-k latest), or of a particular subversion (-k 1.20.+), in which case it'll query GCS
to find available versions, use the list command, to find installed ones use either list or list-installed
this is in pure Go, with no external tool dependencies -- it'll work regardless of shell, tooling installed, etc.

DirectXMan12 · 2021-04-21T08:22:47Z

Other notes: this currently writes the archive to a temp file then reads that back, and cleans it up at the end.. Not strictly necessary -- we could stream directly to the gzip -> tar readers, but it theoretically could make debugging easier. 🤷‍♀️

estroz · 2021-04-21T16:10:53Z

tools/setup-envtest/versions/types.go

+}
+
+// IsWildcard checks if any part of this version is a wildcard.
+func (v Version) IsWildcard() bool {


Rather than implement a bespoke pseudo-semver system, why not use semver with the following rules:

X.Y.Z matches an exact version.

X.Y matches latest patch when downloading/switching, or all patches when cleaning.

You could also pin the meaning of X.Y to latest patch, and pass --all-patches to clean and list commands to get wildcard behavior.

I will say though, the wildcard grammar is not complex.

I wanted a quick way to distinguish "get me the latest" from "get me anything from disk if I have it, otherwise download", but you may be right that it's easier. I'll play around a bit and see if I can find something that works nicely

also, FWIW: all semver selectors are bespoke as far as I can tell -- there doesn't appear to be a spec for the selectors, just the version itself.

DirectXMan12 · 2021-04-21T22:12:00Z

refactoring a bit too enable better no-internet support, and to combine commands, etc

camilamacedo86 · 2021-04-21T22:29:09Z

tools/setup-envtest/README.md

+not guarnateed, except that the leaf directory will contain the names
+expected by envtest.  You should always use `setup-envtest fetch` or
+`setup-envtest switch` (generally with the `-p path` or `-p env` flags) to
+get the directory that you should use.


Would not first be required to move forward with #1234?

Is this one working if:

We remove the bins from: /usr/local/kubebuilder/bin

Then, use it to install the binaries in the bin directory of the project as we do in KB.

Note that currently on the KB side we can just run the tests with make test and not directly because of it. (https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v3/Makefile#L56)

This is an alternate approach to installing in GOBIN. We could, for instance, follow up with something along those lines where we have a well-known symlink in the store directory, and have use or switch change that instead of having you fiddle with the environment variable, but I figured this was a decent first step that's backwards compatible (all CR versions support using $KUBEBUILDER_ASSETS)

estroz · 2021-04-22T19:13:54Z

tools/setup-envtest/remote/client.go

+//
+// The results are sorted with newer versions first.
+func (c *Client) ListVersions(ctx context.Context) ([]versions.Set, error) {
+	loc := &url.URL{


Can you not use the google storage API here?

didn't want to pull in a whole bunch of deps for basically one API call, plus testing is a bit easier.

I can switch it out if you want, but it's not really much code.

Nah that's a fair point.

DirectXMan12 · 2021-04-23T04:44:49Z

Added some tests, made the version selector more semver like:

1.6.3: exact
1.6.x or 1.6.* or just 1.6: any 1.6 (copied from a couple of the existing semver libraries)
~1.6.3: usual meaning (pin on 1.6, 1.6.3 and up)
<1.17: older than 1.17 (useful for cleanup, mainly)
1.6.x!: no really, get the latest remote version
~~1.22.0?: include prerelease versions when searching~~

We can't really use an existing semver library because one of them (blang) is missing support for '~' (maybe not strictly needed?) and the other (Masterminds) has no easy way to construct a version from parts except by round tripping it through a string. We need to keep our own version type because blang's version isnt hashable, and masterminds' is technically hashable but is opaque so it's not guaranteed to stay that way.

EDIT: I realize I mis-read what you were suggesting and I guess technically made this more complicated, but at least it's a bit more familiar? And the < selector is nice for cleanup.

DirectXMan12 · 2021-04-23T04:47:21Z

/retest

DirectXMan12 · 2021-04-23T04:55:41Z

I might simplify the selector logic down a bit closer to what you suggested above tomorrow -- right now we handle a lot of cases that don't currently apply (they could in the future, which is why I added them, but 🤷‍♀️ )

EDIT: simplified the actual code and schema a bit, updated above

tools/setup-envtest/env/env.go

tools/setup-envtest/main.go

This introduces an envtest binary manager tool. It can download binaries from GCS, list available versions, print out help for switching between them, and remove old ones. By default, it downloads binaries into an OS-specific cache directory, as per the OS's conventions, but this can be overridden.

We can now drop the `setup-envtest.sh` script, and instead just use `source <(setup-envtest fetch -k ${version} -p env)` (plus a couple lines to install setup-envtest ;-)).

estroz

/lgtm

k8s-ci-robot · 2021-04-30T22:22:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DirectXMan12, estroz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [DirectXMan12]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Makes use of the new setup-envtest tool (kubernetes-sigs/controller-runtime#1488) in hack/setup-envtest.sh instead of vendoring hack/setup-envtest.sh and fetching binaries with that.

* Upgrade to k/*@v0.21.1 in go.mod * [automated] make revendor for k/* dependencies This deletes pkg/mock/client-go/kubernetes/mocks.go to resolve the following deadlock: make revendor fails because of some dependencies of the file and make generate fails because of missing revendoring. File will be generated again in the next commit. * [automated] make generate for k/* dependencies * Upgrade to c-r@v0.9.0 in go.mod * [automated] make revendor for c-r dependency `make revendor` results in `hack/setup-envtest.sh` being broken, so reset the file after running `make revendor`. Adaption to breaking changes in the upstream file will be done in a later commit. * manager.NewClientBuilder was removed in favor of cluster.DefaultNewClient ref kubernetes-sigs/controller-runtime#1409 * client.*MergeFrom* now take client.Object instead of runtime.Object ref kubernetes-sigs/controller-runtime#1395 * [automated] make generate for c-r dependency * Adapt to changes in labels.NewRequirement ref kubernetes/kubernetes#97538 * Adapt to new setup-envtest tool Makes use of the new setup-envtest tool (kubernetes-sigs/controller-runtime#1488) in hack/setup-envtest.sh instead of vendoring hack/setup-envtest.sh and fetching binaries with that. * [automated] make revendor for setup-envtest tool * Adapt pkg/envtest to upstream changes - Make use of the new Users concept in envtest to provision a dedicated user for gardener-apiserver and a valid kubeconfig - Make use of the new way to configure API server args to easily configure kube-aggregator flags - Also generate certs for aggregation layer on our own instead of reusing the API server ca/certs (which is semantically correct), which allows us to drop our fork including kubernetes-sigs/controller-runtime#1449 * Styling nits

Upstream has replaced the shell script with a single binary. See kubernetes-sigs/controller-runtime#1488 for more information.

* Upgrade to k/*@v0.21.1 in go.mod * [automated] make revendor for k/* dependencies This deletes pkg/mock/client-go/kubernetes/mocks.go to resolve the following deadlock: make revendor fails because of some dependencies of the file and make generate fails because of missing revendoring. File will be generated again in the next commit. * [automated] make generate for k/* dependencies * Upgrade to c-r@v0.9.0 in go.mod * [automated] make revendor for c-r dependency `make revendor` results in `hack/setup-envtest.sh` being broken, so reset the file after running `make revendor`. Adaption to breaking changes in the upstream file will be done in a later commit. * manager.NewClientBuilder was removed in favor of cluster.DefaultNewClient ref kubernetes-sigs/controller-runtime#1409 * client.*MergeFrom* now take client.Object instead of runtime.Object ref kubernetes-sigs/controller-runtime#1395 * [automated] make generate for c-r dependency * Adapt to changes in labels.NewRequirement ref kubernetes/kubernetes#97538 * Adapt to new setup-envtest tool Makes use of the new setup-envtest tool (kubernetes-sigs/controller-runtime#1488) in hack/setup-envtest.sh instead of vendoring hack/setup-envtest.sh and fetching binaries with that. * [automated] make revendor for setup-envtest tool * Adapt pkg/envtest to upstream changes - Make use of the new Users concept in envtest to provision a dedicated user for gardener-apiserver and a valid kubeconfig - Make use of the new way to configure API server args to easily configure kube-aggregator flags - Also generate certs for aggregation layer on our own instead of reusing the API server ca/certs (which is semantically correct), which allows us to drop our fork including kubernetes-sigs/controller-runtime#1449 * Styling nits

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 21, 2021

k8s-ci-robot requested review from alexeldeib and gerred April 21, 2021 07:15

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 21, 2021

k8s-ci-robot assigned camilamacedo86 and joelanford Apr 21, 2021

DirectXMan12 force-pushed the feature/envtest-setup-tool branch from e19b89a to 6ae784e Compare April 21, 2021 07:36

DirectXMan12 force-pushed the feature/envtest-setup-tool branch 2 times, most recently from 11ac855 to 1326169 Compare April 21, 2021 07:53

DirectXMan12 requested review from joelanford and camilamacedo86 and removed request for gerred and alexeldeib April 21, 2021 08:24

estroz reviewed Apr 21, 2021

View reviewed changes

camilamacedo86 reviewed Apr 21, 2021

View reviewed changes

estroz reviewed Apr 22, 2021

View reviewed changes

DirectXMan12 force-pushed the feature/envtest-setup-tool branch 5 times, most recently from 0cba0b5 to 8d7c266 Compare April 23, 2021 04:41

DirectXMan12 force-pushed the feature/envtest-setup-tool branch from 8d7c266 to 614a0ec Compare April 23, 2021 22:36

estroz reviewed Apr 27, 2021

View reviewed changes

tools/setup-envtest/env/env.go Outdated Show resolved Hide resolved

estroz reviewed Apr 27, 2021

View reviewed changes

tools/setup-envtest/main.go Show resolved Hide resolved

DirectXMan12 force-pushed the feature/envtest-setup-tool branch 2 times, most recently from 4c19778 to 334c702 Compare April 28, 2021 00:29

DirectXMan12 added 2 commits April 27, 2021 17:56

Switch hack to use the new envtest tool

782807d

We can now drop the `setup-envtest.sh` script, and instead just use `source <(setup-envtest fetch -k ${version} -p env)` (plus a couple lines to install setup-envtest ;-)).

DirectXMan12 force-pushed the feature/envtest-setup-tool branch from 334c702 to 782807d Compare April 28, 2021 00:56

DirectXMan12 mentioned this pull request Apr 29, 2021

Install and looking for envtest bins in the $GOBIN/ by default #1234

Closed

estroz approved these changes Apr 30, 2021

View reviewed changes

k8s-ci-robot assigned estroz Apr 30, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 30, 2021

k8s-ci-robot merged commit 2c238de into kubernetes-sigs:master Apr 30, 2021

k8s-ci-robot added this to the v0.9.x milestone Apr 30, 2021

burmanm mentioned this pull request May 2, 2021

"make test" target is broken operator-framework/operator-sdk#4856

Closed

DirectXMan12 deleted the feature/envtest-setup-tool branch May 4, 2021 22:00

schrodit mentioned this pull request May 27, 2021

Use new EnvTest Binaries Setup Tool gardener/landscaper#208

Closed

thiscantbeserious mentioned this pull request Jun 8, 2021

setup-envtest has been changed in master (script removed) seaweedfs/seaweedfs-operator#39

Open

stefanprodan mentioned this pull request Jun 10, 2021

Update kubebuilder to 3.1.0 fluxcd/pkg#108

Closed

timebertt mentioned this pull request Jun 16, 2021

Upgrade sigs.k8s.io/controller-runtime to v0.9, k8s.io/* to v0.21 gardener/gardener#4174

Closed

BostjanBozic mentioned this pull request Jun 24, 2021

Scaffolding Makefile using controller-runtime v0.9.1 kubernetes-sigs/kubebuilder#2235

Closed

horis233 mentioned this pull request Aug 19, 2021

build: bump k8s dependencies to 1.21 IBM/operand-deployment-lifecycle-manager#736

Merged

alekc mentioned this pull request Sep 6, 2021

Local Development Guidelines Update external-secrets/external-secrets#95

Closed

raghavendra-talur mentioned this pull request Sep 13, 2021

replace the setup-envtest shell script with the new binary red-hat-storage/odf-multicluster-orchestrator#41

Closed

astefanutti mentioned this pull request Dec 7, 2023

[test] Upgrade envtest to latest version ray-project/kuberay#1720

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ EnvTest Binaries Setup Tool #1488

✨ EnvTest Binaries Setup Tool #1488

DirectXMan12 commented Apr 21, 2021 •

edited

DirectXMan12 commented Apr 21, 2021

DirectXMan12 commented Apr 21, 2021 •

edited

DirectXMan12 commented Apr 21, 2021 •

edited

estroz Apr 21, 2021

DirectXMan12 Apr 21, 2021

DirectXMan12 Apr 23, 2021

DirectXMan12 commented Apr 21, 2021 •

edited

camilamacedo86 Apr 21, 2021 •

edited

DirectXMan12 Apr 23, 2021

estroz Apr 22, 2021

DirectXMan12 Apr 23, 2021

DirectXMan12 Apr 23, 2021

estroz Apr 23, 2021

DirectXMan12 commented Apr 23, 2021 •

edited

DirectXMan12 commented Apr 23, 2021

DirectXMan12 commented Apr 23, 2021 •

edited

estroz left a comment

k8s-ci-robot commented Apr 30, 2021

✨ EnvTest Binaries Setup Tool #1488

✨ EnvTest Binaries Setup Tool #1488

Conversation

DirectXMan12 commented Apr 21, 2021 • edited

DirectXMan12 commented Apr 21, 2021

DirectXMan12 commented Apr 21, 2021 • edited

DirectXMan12 commented Apr 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DirectXMan12 commented Apr 21, 2021 • edited

camilamacedo86 Apr 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DirectXMan12 commented Apr 23, 2021 • edited

DirectXMan12 commented Apr 23, 2021

DirectXMan12 commented Apr 23, 2021 • edited

estroz left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 30, 2021

DirectXMan12 commented Apr 21, 2021 •

edited

DirectXMan12 commented Apr 21, 2021 •

edited

DirectXMan12 commented Apr 21, 2021 •

edited

DirectXMan12 commented Apr 21, 2021 •

edited

camilamacedo86 Apr 21, 2021 •

edited

DirectXMan12 commented Apr 23, 2021 •

edited

DirectXMan12 commented Apr 23, 2021 •

edited