New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1 minute slower since kubernets version 1.20.3 - [kubelet-check] Initial timeout of 40s passed. #2395
Comments
hi, @medyagh
this indicates the kubelet takes a while to become ready. i don't think we have anything on the kubeadm side that can cause it, so it might be a core issue. |
i can confirm the problem in our CI (kind(er) based doing containerd-in-docker in a prow pod): this is using a 1.20.4 kubeadm with a 1.20.4 kubelet.
this is the whole diff of k/k between 1.20.2 and 1.20.3: there are kubeadm commits that change the default image repository for CI images, which doesn't seem related.
i will now try a 1.20.3 kubeadm with a 1.20.2 kubelet and bisect. |
this is also a problem in the tip of 1.21 and 1.19:
so we must have backported something that causes this. looking at logs and diffs this started happening on the 13th of Jan. my current suspect commit is: |
@SergeyKanzhelev @ehashman please see a possible regression |
before kubernetes/kubernetes@88ee6d7
after:
the PR that did this is: so i can see this wait making sense, but it should bail out if this is the first node and there is no API server yet. |
cc @BenTheElder for kind |
i have commented on the original PR. |
We bumped apiserver timeout till 3 mins when commited 4da07ce one but looks like CI is still failing when testing 4.7.0-rc.3 bits. Upstream issue: kubernetes/kubeadm#2395
for a self-hosting kubelet, informer sync will fail until the API server is available. Does the delay mean that it takes 40 seconds to start the self-hosted API server? or does it mean that GetNode() is called 4 times and times out in the process of starting static pods? |
i suspect that something like that is happening. the base of the time comparison is a check in kubeadm where it asks both the kubelet and the apiserver to report 200 on their after this change the kubelet |
So just to be clear, the apiserver still starts up just as quickly, it's just that the kubelet now does not return 200 on healthz until the apiserver is also up? |
seems that both the kubelet and api-server healthz are delayed. logs from kubeadm:
|
i have a WIP proposal fix for the kubelet. |
cc @adisky |
I haven't noticed any substantial delays in cluster creation on Kubernetes since this patch went in, is it possible this is minikube-specific and should be fixed on minikube bootstrap? The patch in kubernetes/kubernetes#94087 is a correctness fix to avoid nodes admitting pods before they have fully synced pod state. We've seen many issues with pods getting stuck in NodeAffinity status after cold-starting clusters without this patch because the kubelet starts admitting pods before it is ready and .
I don't think the kubelet has any way of knowing it's the first node? Are you suggesting a change in k8s or kubeadm? Edit: oops I thought I hit send on this comment 3h ago but apparently not 😅 |
what cluster creation tooling / e2e test jobs are you checking this against?
this patch definitely creates a problem if the api-server is managed as a static pod.
i do not disagree with the intent, i just think the implementation can be improved. |
fix attempt at: |
We bumped apiserver timeout till 3 mins when commited 4da07ce one but looks like CI is still failing when testing 4.7.0 bits. Upstream issue: kubernetes/kubeadm#2395
We bumped apiserver timeout till 3 mins when commited 4da07ce one but looks like CI is still failing when testing 4.7.0 bits. Upstream issue: kubernetes/kubeadm#2395
What I'm confused by is why this would affect bootstrap at all. For a standalone kubelet, the checks are always bypassed: https://github.com/kubernetes/kubernetes/pull/94087/files#diff-67dd9e8f3ee257072765326cb4f242852554a2c0753563fa51e292c0a63a7b94R455-R460
OpenShift and upstream Kubernetes. I don't have specific jobs to point you to though, idk if there's an easy way to gather stats on cluster start time from testgrid. |
"self hosted" is not always standalone. kubeadm sets up a kubelet which runs the apiserver and then registers itself as a node to the API server it started |
In particular for development clusters like kind/minikube etc. using
But as Jordan mentioned even kubeadm clusters with N nodes have static pod apiserver on certain nodes + registered kubelets for all nodes. This will slow down bootstrap for pretty much any kubeadm cluster since most work is blocked on healthy API server / first node.
There is data of sorts, just not what you want probably. Jobs running under kubetest (GCE) record the Local cluster jobs (local-up-cluster.sh, kind) are currently impacted by CI/prow.k8s.io I/O throttling issues impacting pod startup, build times, etc. badly in general and wouldn't make useful performance data until that's resolved. |
Our RKE2 project does the same thing. Runs etcd plus apiserver and sundry as static pods; the kubelet registers to the local apiserver once it's up. |
FYI, got pinged about this issue by some Cluster API users... |
We bumped apiserver timeout till 3 mins when commited 4da07ce one but looks like CI is still failing when testing 4.7.0 bits. Upstream issue: kubernetes/kubeadm#2395
update PR in master (1.22) merged: the change is backported all the way back to 1.18 (inc); backport PRs are pending. |
PRs merged. |
(i guess the 1.21 PR has not merged yet, but is about to.) you can expect new releases after the 12th of May. |
### What changes were proposed in this pull request? This PR aims to upgrade the minimum Minikube version to 1.18.0 to 1.28.0 for Apache Spark 3.4.0. ### Why are the changes needed? Minikube v1.28.0 is released on `Nov 4th, 2022` and the latest one as of today. - https://github.com/kubernetes/minikube/releases/tag/v1.28.0 GitHub Action CI has been using `Minikube 1.28.0` to test `K8s v1.25.3` and Homebrew also provides `1.28.0` by default. - https://github.com/apache/spark/actions/runs/3681318787/jobs/6227888255 ``` * minikube v1.28.0 on Ubuntu 20.04 ... * Downloading Kubernetes v1.25.3 preload ... ``` In addition, we can choose different K8s versions on this latest Minikube like the following. ``` $ minikube start --kubernetes-version=1.21.0 😄 minikube v1.28.0 on Darwin 13.1 (arm64) ❗ Kubernetes 1.21.0 has a known performance issue on cluster startup. It might take 2 to 3 minutes for a cluster to start. ❗ For more information, see: kubernetes/kubeadm#2395 ✨ Automatically selected the docker driver 📌 Using Docker Desktop driver with root privileges 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 💾 Downloading Kubernetes v1.21.0 preload ... ``` ### Does this PR introduce _any_ user-facing change? No. This is a dev-only change. ### How was this patch tested? Pass the CIs. Closes #39043 from dongjoon-hyun/SPARK-41502. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade the minimum Minikube version to 1.18.0 to 1.28.0 for Apache Spark 3.4.0. ### Why are the changes needed? Minikube v1.28.0 is released on `Nov 4th, 2022` and the latest one as of today. - https://github.com/kubernetes/minikube/releases/tag/v1.28.0 GitHub Action CI has been using `Minikube 1.28.0` to test `K8s v1.25.3` and Homebrew also provides `1.28.0` by default. - https://github.com/apache/spark/actions/runs/3681318787/jobs/6227888255 ``` * minikube v1.28.0 on Ubuntu 20.04 ... * Downloading Kubernetes v1.25.3 preload ... ``` In addition, we can choose different K8s versions on this latest Minikube like the following. ``` $ minikube start --kubernetes-version=1.21.0 😄 minikube v1.28.0 on Darwin 13.1 (arm64) ❗ Kubernetes 1.21.0 has a known performance issue on cluster startup. It might take 2 to 3 minutes for a cluster to start. ❗ For more information, see: kubernetes/kubeadm#2395 ✨ Automatically selected the docker driver 📌 Using Docker Desktop driver with root privileges 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 💾 Downloading Kubernetes v1.21.0 preload ... ``` ### Does this PR introduce _any_ user-facing change? No. This is a dev-only change. ### How was this patch tested? Pass the CIs. Closes apache#39043 from dongjoon-hyun/SPARK-41502. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Hello, on minikube we updated our default kubernetes to v1.20.4 and we noticed bumping from v1.20.2 to v1.20.3 causes 1 minute slow down on miikube which the logs shows it is in kubeadm init
kubernetes/minikube#10545
is there a new config or somethign that we are missing out int he v1.20.3 that is adding an additional check that causes 1 minute tax ?
screenshot of our performance dashboard
The text was updated successfully, but these errors were encountered: