RKE - Add support for provisioning 1.24 clusters

[jiaqiluo]

K8S 1.24 was released: https://github.com/kubernetes/kubernetes/releases/tag/v1.24.0
We need to add the support for provisioning 1.24 clusters in RKE

TODO:

• add the new k8s template to the KDM
  • review and update add-on components versions
    • etcd 3.5.4
    • coredns 1.9.3
  • bring the new templates into RKE
• bump the go-client and other modules used in RKE to the versions that support k8s 1.24
• make sure we can upgrade the existing cluster from 1.23 to 1.24. One known caveat related to the CNI: https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/#cni-version-related-breaking-change
• set enable_cri_dockerd to true by default for k8s >= 1.24 Add support for Dockershim rancher#31943 (comment)
• evaluate performance issues with cri-dockerd on 1.22/1.23 cluster with enable_cri_dockerd flag turned on/off per RKE - Add support for provisioning 1.24 clusters #2936 (comment) (note that as of 5/16/22 our fork is 62 commits behind upstream, so moving to the latest version may be beneficial); related issue - cri_dockerd as included in rke-tools has extremely bad performance #2938

SURE-3940

[chfrank-cgn]

The version of cri_dockerd in RKE1 1.23.6 (as bundled in Rancher 2.6.5) has extremely bad performance; just installing an empty RKE1 cluster will max out a 4vCPU VM.

[snasovich]

Mirantis/cri-dockerd#38 may be related and help address the performance issues.

[jiaqiluo]

Mirantis/cri-dockerd#38 may be related and help address the performance issues.

In order to use the new version of cri-dockerd, we need to

1. update https://github.com/rancher/cri-dockerd/ to get the upstream's change , tag a new release update: because the upstream is actively maintained now, we will use the upstream's releases directly instead of building it ourselves
2. update the version in https://github.com/rancher/rke-tools/blob/ef0a37a04adf865c50f0d115ec0efcd059c7a907/package/Dockerfile#L8 and tag a new image
3. use the new rke-tool image tag in KDM for 1.24

#2938 is tracking the progress for this problem

[jiaqiluo]

PR to mirror images: rancher/image-mirror#254

[jiaqiluo]

can be tested on v1.3.13-rc2 https://github.com/rancher/rke/releases/tag/v1.3.13-rc2

At least to test:

• provision a new 1.24 cluster
• upgrade from a previous version to 1.24
• set/ unset the enable_cri_dockerd filed in the cluster.yml file:
  • note:it is enabled by default when the cluster version >= 1.24 AND the option enable_cri_dockerd is not configured in the cluster.yml file

[Josh-Diamond]

Test cases and results were tracked and may be found in confluence under:

• Rancher QA > Test Plans > Kubernetes Version Support > RKE > K8s 1.24 support

[Josh-Diamond]

Reported testing blockers:

• Unable to create k8s v1.24 downstream clusters using Oracle Linux 8.4 rancher#38214 - Set for 2.7
• Pods getting stuck into "Terminating" state on deletion after the cluster is upgraded to k8s v1.24 rancher#38270 - Set to be release-noted
• When provisioning an RKE1 downstream cluster with io.cattle.cluster.cridockerd.enable set to false and enable_cri_dockerd set to true, the cluster successfully provisions when it shouldn't rancher#38265 - Set for 2.7