Tasks to bump to Kubernetes v1.28

[ykakarap]

This issue is tracking the tasks that should be implemented after the Kubernetes minor release has been released.

Tasks

Prerequisites:

• Decide which Cluster API release series will support the new Kubernetes version
  • If feasible we usually cherry-pick the changes back to the latest release series.
  • => As usuall will implement on main and cherry-pick into release-1.5

Supporting managing and running on the new Kubernetes version

This section contains tasks to update our book, e2e testing and CI to use and test the new Kubernetes version
as well as changes to Cluster API that we might have to make to support the new Kubernetes version. All of these
changes should be cherry-picked to all release series that will support the new Kubernetes version.

• Modify quickstart and CAPD to use the new Kubernetes release: ✨ v1.28: Prepare quickstart, capd and tests for the new release including kind bump #9160
  • Bump the Kubernetes version in:
    • test/*: search for occurrences of the previous Kubernetes version
    • Tiltfile
  • Ensure the latest available kind version is used (including the latest images for this kind release)
  • Verify the quickstart manually
  • Prior art: ⚠️ Use Kubernetes 1.25 in Quick Start docs and CAPD. #7156
  • bump InitWithKubernetesVersion and WorkloadKubernetesVersion in clusterctl_upgrade_test.go
• Ensure the jobs are adjusted to provide test coverage according to our support policy: CAPI: bump jobs for v1.28 kubernetes/test-infra#30347
  • For the main branch and the release branch of the latest supported Cluster API minor release:
    • Add new periodic upgrade job.
    • Adjust presubmit jobs so that we have the latest upgrade jobs available on PRs.
  • For the main branch:
    • periodics & presubmits:
      • Bump KUBEBUILDER_ENVTEST_KUBERNETES_VERSION of the test-mink8s jobs to the new minimum supported management cluster version.
    • periodics:
      • Bump KUBERNETES_VERSION_MANAGEMENT of the e2e-mink8s job to the new minimum supported management cluster version.
      • Drop the oldest upgrade job as the oldest Kubernetes minor version is now out of support.
  • Prior art: Bump cluster-api jobs for v1.25/v1.26 kubernetes/test-infra#27421
• Update book: 📖 book: Update versions doc adding Kubernetes 1.28 + prowjob documentation #9161
  • Update supported versions in versions.md
  • Update job documentation in jobs.md
  • Prior art: 🌱 Update versions doc adding Kubernetes 1.25 #7194 📖 docs update jobs.md #7196
• Issues specific to the Kubernetes minor release:
  • Sometimes there are adjustments that we have to make in Cluster API to be able to support
    a new Kubernetes minor version. Please add these issues here when they are identified.
  • Check and update the Kubernetes version skew policy in the MachineSet preflight checks. ✨ machineset: adjust preflight check to allow kubelet version skew of 3 for clusters running v1.28 and above #9222
    • More details: ✨ MS preflight checks to improve cluster stability #8595 (comment)

Using new Kubernetes dependencies

This section contains tasks to update Cluster API to use the latest Kubernetes Go dependencies and related topics
like using the right Go version and build images. These changes are only made on the main branch. We don't
need them in older releases as they are not necessary to manage workload clusters of the new Kubernetes version or
run the Cluster API controllers on the new Kubernetes version.

• Ensure there is a new controller-runtime minor release which uses the new Kubernetes Go dependencies.
• Update our Prow jobs for the main branch to use the correct kubekins-e2e image Cluster API: bump presubmits to kubekins 1.28 kubernetes/test-infra#30461 Cluster API: bump periodics to kubekins 1.28 kubernetes/test-infra#30462
  • It is recommended to have one PR for presubmit and one for periodic jobs to reduce the risk of breaking the periodic jobs.
  • Prior art: presubmit jobs: Upgrade cluster-api kubekins-e2e version (v20220829-d5650c79bf-1.24 -… kubernetes/test-infra#27311
  • Prior art: periodic jobs: Upgrade cluster-api kubekins-e2e version for periodics-main (v2022082… kubernetes/test-infra#27326
• Bump the Go version in Cluster API: (if Kubernetes is using a new Go minor version) (does not apply)
  • Search for the currently used Go version across the repository and update it
  • We have to at least modify it in: hack/ensure-go.sh, .golangci.yml, cloudbuild*.yaml, go.mod, Makefile, netlify.toml, Tiltfile
  • Prior art: 🌱 Upgrade golang version (1.18.3 -> 1.19.0) #7135
• Bump controller-runtime ⚠️ Bump to controller-runtime v0.16 #8999
• Bump controller-tools 🌱 Bump controller tools to v0.13.0 #9221
• Bump the Kubernetes version used in integration tests via KUBEBUILDER_ENVTEST_KUBERNETES_VERSION in Makefile 🌱 Bump envtest binaries to 1.28 #9268
  • Note: This PR should be cherry-picked as well. It is part of this section as it depends on kubebuilder/controller-runtime
    releases and is not strictly necessary for Supporting managing and running on the new Kubernetes version.
  • Prior art: 🌱 Update kubebuilder envtest (1.24.2 -> 1.25.0) #7193
• Bump conversion-gen via CONVERSION_GEN_VER in Makefile 🌱 Bump conversion-gen to v0.28 #9267
  • Prior art: 🌱 Update conversion-gen version (v0.23.1 -> v0.25.0) #7118

[killianmuldoon]

/triage accepted

[fabriziopandini]

Thanks for starting planning work for the intake of the upcoming K8s release!
(note, planned release date as of today is Aug 15th)

[sbueringer]

We mostly opened this issue already so we can start collecting release-specific tasks "somewhere". If someone has other release-specific tasks, please let us know :)

[sbueringer]

/assign

I've already started working on supporting v1.28 in CR

[chrischdi]

/assign

After talking to @sbueringer : I'll takeover the Supporting managing and running on the new Kubernetes version part for now.

[furkatgofurov7]

@sbueringer @chrischdi do you need help with this issue since it is in parts(tracking one)? If so, please let me know and we can distribute it as we did in the prev release cycle :)

[sbueringer]

Thx, I think we're good for now. IIRC Christian should have most of it already locally. For the second section I have to wait for the CR release

(And we can't merge the PRs for the first section before the Kubernetes 1.28 release next week)

[sbueringer]

Would be really good if we can finish #9097 so PRs building on top are not getting too messy (merge conflicts etc)

[chrischdi]

I pre-created PRs for the Supporting managing and running on the new Kubernetes version part.

They should get merged in order and the first one depends on #9097 .

Also the first one depends on the v1.28 release getting done and on getting a kind release.

They are all marked as WIP and draft for now.

Supporting managing and running on the new Kubernetes version

• Modify quickstart and CAPD to use the new Kubernetes release:

  • ✨ v1.28: Prepare quickstart, capd and tests for the new release including kind bump #9160

• Ensure the jobs are adjusted to provide test coverage according to our support policy:

  • CAPI: bump jobs for v1.28 kubernetes/test-infra#30347

• Update book:

  • 📖 book: Update versions doc adding Kubernetes 1.28 + prowjob documentation #9161

[sbueringer]

All 3 PRs are merging now. @chrischdi PTAL at cherry-picking the two non-test-infra PRs into release-1.5. I suspect they might need manual cherry-picks

[sbueringer]

So first half is basically done (+/- keeping an eye on testgrid for a bit). The second depends on the CR release

[furkatgofurov7]

@sbueringer is there a release timeline for CR release?

[sbueringer]

No, probably a few days. But we should have > 2 months time for it in CAPI

[sbueringer]

CR v0.16 was released today. Opened a few PRs for the second section:

• test-infra:
  • Cluster API: bump presubmits to kubekins 1.28 kubernetes/test-infra#30461
  • Cluster API: bump periodics to kubekins 1.28 kubernetes/test-infra#30462
• bump CR
  • ⚠️ Bump to controller-runtime v0.16 #8999
• bump conversion-gen
  • 🌱 Bump conversion-gen to v0.28 #9267
• bump envtest binaries:
  • 🌱 Bump envtest binaries to 1.28 #9268
    • waiting on a release of the kubebuilder tools for Kubernetes 1.28 (https://kubernetes.slack.com/archives/CAR30FCJZ/p1692723835465719)

I think otherwise we should be done. Would be great if someone can double-check if I/we missed something (@killianmuldoon @chrischdi)

[furkatgofurov7]

I think otherwise we should be done. Would be great if someone can double-check if I/we missed something

Double checked it and looks like everything is covered with above PRs 👍🏼

[sbueringer]

@furkatgofurov7 thank you!

[sbueringer]

Remaining PRs

• bump envtest binaries: 🌱 Bump envtest binaries to 1.28 #9268
  • depends on ✨ Build kubebuilder-tools for k8s 1.28.0 kubebuilder#3545

[kinarashah]

@sbueringer do you know timeline for when this bump to k8s 1.28 would be released?

[sbueringer]

Next Tuesday

[furkatgofurov7]

@sbueringer looks like we can close this issue?

[sbueringer]

Yup

/close

[k8s-ci-robot]

@sbueringer: Closing this issue.

In response to this:

Yup

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.