Tracking: Upstream etcd v3.5 issues

[phillipsj]

Describe the bug
Upstream etcd posted that there are issues with v3.5 and that everyone should stay on 3.4.x and only upgrade once 3.5.3 is released.

https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ

[rosskirkpat]

The upstream issue has been closed and a new release of etcd, v3.5.3, is being tracked here

There is a fix in upstream etcd merged in main and a backport to the release-3.5 branch.

We should prepare for an etcd bump once this release is out (currently unknown ETA). However, the issue with data inconsistency will not necessarily be fixed in it's entirety in the 3.5.3 release per this comment

It would be great to have automatic way to prevent data inconsistencies in the future, however this is not required immediate release.

[rosskirkpat]

cc: @deniseschannon @Jono-SUSE-Rancher @kinarashah @sirredbeard

It looks like the upcoming v3.5.3 release of etcd will not have a full fix for the data inconsistency issue. We should discuss what our plan looks like for 2.6.5

Please see the comment below on clarification. v3.5.3 will fully fix the data inconsistency issue.

[serathius]

Sorry for confusion in wording of the message you quoted, v3.5.3 will fully the data inconsistency issue. What we are not planning is automatic testing/reproduction as this will be done manually.

[rosskirkpat]

Thank you for the clarification @serathius! That's great to hear.

[kinarashah]

etcd bump to v3.5.3 is getting updated as a part of #37388 for RKE1 clusters.

[snasovich]

Closing this issue as the versions with fixed etcd version are now available in Rancher following the latest KDM update.