Proposal: test kubernetes pre-releases against minikube

[medyagh]

in minikube we have made several performance measuring tools both for measuring the latency and also resource usage, that gets tested with minikube, I would like to test every alpha release of kuberentes with minikube, preferably on the kubernete side. (prow)

previous years minikube could have not be tested in prow because it was only VM drivers, we have support for docker/podman since 2019, I would love to start testing kuberentes with minikube to avoid performance surprises like this :

related issues:
kubernetes/minikube#10545
kubernetes/kubeadm#2395 (comment)

the Minikube maintainers (cc @medyagh) [found](https://github.com/kubernetes/kubeadm/issues/2395) that this change introduced a performance regression.
when client credentials are fed to a primary kubelet in a cluster (`--kubeconfig`), but there are no other nodes or an API server running yet (because this primary kubelet will boot it from a static pod), this blocks for extra ~40 seconds.

i can see the informer wait as something desired but i think the logic here can be improved.
cc @derekwaynecarr @deads2k @liggitt 

we also backported this change in the support skew and since we don't have direct performance tests, nobody saw it until now.

Originally posted by @neolit123 in #94087 (comment)

[k8s-ci-robot]

@medyagh: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

@BenTheElder proposed that we start timing the kind deployments which are already in the e2e here and use kubeadm. these jobs run in high-priority and we can catch perf. problems with them.

both kind and minikube are using kubeadm, so could also have the timing tests in the kubeadm e2e, but the kubeadm e2e are currently running in low-priority.

/sig testing cluster-lifecycle
/kind feature
/area test

[medyagh]

could we add minikube two kubedm e2e too ? adding minikube tests would add testing for crio,docker in addition to containerd runtime and we could test the minikube's functional test against each rest release

[neolit123]

the kubeadm e2e test setup is based on something that resembles kind (uses it as a library).

i can see you already have prow jobs for the minikube repository:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/minikube/minikube.yaml

you could add more periodic and pre/post-submit jobs there.

[medyagh]

excellent ! thank you for link, @azhao155 on minikube side is interesting in helping with this. I will add it our v1.19.0 milestone

[BenTheElder]

I would like to test every alpha release of kuberentes with minikube, preferably on the kubernete side. (prow)

Anyone can add jobs to CI, however there are documented requirements for signal the release team monitors:
https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md

There is no guarantee that the CI signal team agrees to monitor any given signal, however the minimum requirements are listed there.

Note also that there's no "test on every alpha release" CI functionality. There is "test on merge" and "test periodically" (cron).
Releases are soft-blocked on certain jobs passing reliably within that branch, as inspected by the release team. Though primarily the 1.X release.

[...] adding minikube tests would add testing for crio,docker [..]

Kubernetes does not try to test against the matrix of all integrations nor do we wish to, this is the burdern of the implementers of these integrations (see also: CSI, CNI, we provide an API contract to meet, we do not test everyone that implements it). We need to test against one concrete CRI implementation and are winding down dockershim based CI to a minimum (we have some still) as we plan to remove dockershim. This is incidentally why kind does not support more than one CRI ... (Kubernetes does-not/would-not use this).

proposed that we start timing the kind deployments which are already in the e2e here and use kubeadm. these jobs run in high-priority and we can catch perf. problems with them.

yes, this is something we did not monitor in CI previously because prow did not provide guaranteed QOS anywhere. it does now for high prio jobs (xref the whole CI policy revamp effort), so we can finally add this.

[BenTheElder]

Today we typically performance test on GCE clusters, with Kubernetes SIG Scalability primarily owning this space.

We have performance test presubmits even, but "cluster startup time" is not something the project has chosen to block on so far. If we do choose to block on this, we could easily do so it in our existing test jobs, we're already recording the timing typically.

It's not clear that the project is willing to block on this sort of thing. (Though I personally would welcome it!)

EDIT: One reason not to do this is it's also a property of the provisioning tool, addons etc., and the CI capacity. Most of the time it's not intrinsically coupled to timeout behavior like this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.