1.29-apiserver based aggregated APIServer does not work against 1.28 kube-apiserver #124533
Comments
k8s-ci-robot
added
needs-sig
|
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/sig api-machinery |
k8s-ci-robot
added
sig/api-machinery
|
Has this skew direction (aggregated apiserver newer than kube-apiserver) been supported historically? |
|
That's a good point, when I looked at the official doc we didn't mention it anywhere so I guess that it might just not be supported? Thinking about it again, it would make more sense to me that we would not support this. Feel free to close if we don't support this skew direction |
|
IIUC, at least for Kube components, nothing can be newer than the kube-apiserver (or newer than the oldest kube-apiserver, if there is skew between kube-apiserver instances): https://kubernetes.io/releases/version-skew-policy/. I would feel better about closing if someone else can +1 my understanding. |
|
We can tweak the docs to clarify the situation, whichever way we decide / confirms it goes. |
I think you mean “no component other than the API server can be newer than the API server, and API servers that are not the oldest API server version can be at most one minor version newer than the oldest API server version”. Does that sound right @benluddy? |
|
I'm not sure. The language needs to distinguish between the
And the supported skew between kube-apiserver and the named components in the doc is relative to the version of the oldest kube-apiserver instance. The ambiguity here is with the supported skew between kube-apiserver and any aggregated apiserver based on a particular version of |
liggitt
changed the title
|
Controller clients outside kube-apiserver can't be newer than the kube-apiserver they are talking to (true for kubelet, kube-controller-manager and kube-scheduler, true for aggregated apiservers as well) kube-apiserver supports -1 skew because it talks to itself for APIs it needs for running admission stuff like flowcontrol and webhook/CEL admission, so it is guaranteed to have access to the latest API version. Controllers that talk to kube-apiserver have to wait until kube-apiserver is upgraded to have the same guarantee. We should describe that better in skew docs /remove-kind bug |
k8s-ci-robot
added
kind/documentation
|
This was discussed at length today in the SIG meeting: https://www.youtube.com/watch?v=0TXm-DGcK1k, starting at minute 33:00 |
What happened?
We bumped an aggregated apiserver library based on the generic apiserver library to 1.29 in kubernetes-sigs/custom-metrics-apiserver#170 and users mentioned that after pulling the new dependencies, the aggregated apiservers started failing on Kubernetes 1.28 clusters with the following errors:
What did you expect to happen?
I expected 1.29 libraries to still be compatible with Kubernetes 1.28 based on the kube-apiserver version skew.
How can we reproduce it (as minimally and precisely as possible)?
I haven't tested it with the sample-apiserver yet, but I assume that running its 1.29 version on a 1.28 kind cluster will result in the same failure.
If not, feel free to reach out to me I can share a reproducer with prometheus-adapter, and aggregated apiserver where the problem exists.
Anything else we need to know?
The error appeared with the graduation of P&F to stable in 1.29. I chatted briefly with @tkashem about it and he mentioned that the controller logic was moved from v1beta3 to v1 in 1.29 which is also the version that introduces the v1 API. Meaning that retro compatibility with 1.28 was broken.
The text was updated successfully, but these errors were encountered: