Change k8s default to v1.21

[kinarashah]

There's an upstream issue with etcd 3.5.0-3.5.2, tracked in rancher here: #37157.

k8s v1.22 uses v3.5.0 and v1.22 was set as our default version for Rancher v2.6.4 and RKE v1.3.8, so we want to change the default back to v1.21. This will show 1.22 as experimental in Rancher when creating new clusters and let users know it's not recommended for production.

Rancher local cluster - Docker Install

We cannot change local embedded k3s version used for docker installs without releasing a new rancher version. Requires change in our code and building a new tag. (For example, #37013)

Rancher local cluster - HA

Update Rancher v2.6.4 release notes to reflect upstream recommendations, suggesting not to use v1.22 as local cluster.

RKE1 standalone

Existing clusters provisioned using RKE <= v1.3.8 and k8s v1.22.x will need to add kubernetes_version in cluster.yml before running rke up with RKE v1.3.9. This ensures provisioned clusters don't auto-downgrade. New clusters created with RKE v1.3.9 will be created with k8s v1.21.x by default.

Rancher provisioned RKE1/RKE2/k3s

Existing clusters running v1.22.x won't downgrade, latest v1.22.x will show up as experimental in Edit Cluster if a newer version is available (until a fixed version is released). When creating new clusters, latest available v1.22.x will show as experimental.

[rosskirkpat]

There's an update on the upstream issue, which I summarized here

[snasovich]

This is no longer releavant