Fresh kubernetes deployment using docker runtime - Unable to start api-server and kubelet #9633
Comments
|
@h-182 I think you need to report it to https://github.com/Mirantis/cri-dockerd or kubernetes, and not containerd. |
|
I came across this statement: |
Description
OS: Ubuntu 20.04.6 LTS
Kuberentes:
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
-->
Describe the results you received and expected
kubeadm init --pod-network-cidr=16.10.0.0/8 --apiserver-advertise-address=16.32.12.161 --ignore-preflight-errors=NumCPU,Mem --v=6
I0113 04:33:40.032528 4844 initconfiguration.go:117] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I0113 04:33:40.032686 4844 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I0113 04:33:40.035595 4844 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
I0113 04:33:42.245729 4844 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28
I0113 04:33:42.245807 4844 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.28.txt
[init] Using Kubernetes version: v1.28.5
[preflight] Running pre-flight checks
I0113 04:33:44.898098 4844 checks.go:563] validating Kubernetes and kubeadm version
I0113 04:33:44.898139 4844 checks.go:168] validating if the firewall is enabled and active
I0113 04:33:44.914418 4844 checks.go:203] validating availability of port 6443
I0113 04:33:44.914979 4844 checks.go:203] validating availability of port 10259
I0113 04:33:44.915330 4844 checks.go:203] validating availability of port 10257
I0113 04:33:44.915669 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0113 04:33:44.915821 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0113 04:33:44.915987 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0113 04:33:44.916143 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0113 04:33:44.916351 4844 checks.go:430] validating if the connectivity type is via proxy or direct
I0113 04:33:44.916407 4844 checks.go:469] validating http connectivity to first IP address in the CIDR
I0113 04:33:44.916449 4844 checks.go:469] validating http connectivity to first IP address in the CIDR
I0113 04:33:44.916477 4844 checks.go:104] validating the container runtime
I0113 04:33:44.954410 4844 checks.go:639] validating whether swap is enabled or not
I0113 04:33:44.954552 4844 checks.go:370] validating the presence of executable crictl
I0113 04:33:44.954621 4844 checks.go:370] validating the presence of executable conntrack
I0113 04:33:44.954671 4844 checks.go:370] validating the presence of executable ip
I0113 04:33:44.954722 4844 checks.go:370] validating the presence of executable iptables
I0113 04:33:44.954778 4844 checks.go:370] validating the presence of executable mount
I0113 04:33:44.954822 4844 checks.go:370] validating the presence of executable nsenter
I0113 04:33:44.954867 4844 checks.go:370] validating the presence of executable ebtables
I0113 04:33:44.954913 4844 checks.go:370] validating the presence of executable ethtool
I0113 04:33:44.954956 4844 checks.go:370] validating the presence of executable socat
I0113 04:33:44.954999 4844 checks.go:370] validating the presence of executable tc
I0113 04:33:44.955040 4844 checks.go:370] validating the presence of executable touch
I0113 04:33:44.955084 4844 checks.go:516] running all checks
I0113 04:33:44.967646 4844 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I0113 04:33:44.967745 4844 checks.go:605] validating kubelet version
I0113 04:33:45.000022 4844 checks.go:130] validating if the "kubelet" service is enabled and active
I0113 04:33:45.006196 4844 checks.go:203] validating availability of port 10250
I0113 04:33:45.006243 4844 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0113 04:33:45.006267 4844 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0113 04:33:45.006277 4844 checks.go:203] validating availability of port 2379
I0113 04:33:45.006289 4844 checks.go:203] validating availability of port 2380
I0113 04:33:45.006302 4844 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0113 04:33:45.006374 4844 checks.go:828] using image pull policy: IfNotPresent
I0113 04:33:45.023755 4844 checks.go:854] pulling: registry.k8s.io/kube-apiserver:v1.28.5
I0113 04:34:53.454374 4844 checks.go:854] pulling: registry.k8s.io/kube-controller-manager:v1.28.5
I0113 04:35:46.053991 4844 checks.go:854] pulling: registry.k8s.io/kube-scheduler:v1.28.5
I0113 04:37:02.823926 4844 checks.go:854] pulling: registry.k8s.io/kube-proxy:v1.28.5
W0113 04:37:24.696705 4844 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
I0113 04:37:24.713794 4844 checks.go:854] pulling: registry.k8s.io/pause:3.9
I0113 04:37:30.001247 4844 checks.go:854] pulling: registry.k8s.io/etcd:3.5.9-0
I0113 04:38:51.184189 4844 checks.go:854] pulling: registry.k8s.io/coredns/coredns:v1.10.1
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0113 04:39:33.027558 4844 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0113 04:39:33.122552 4844 certs.go:519] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 16.32.12.161]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0113 04:39:33.314136 4844 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0113 04:39:33.392288 4844 certs.go:519] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0113 04:39:33.699882 4844 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0113 04:39:33.884850 4844 certs.go:519] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com localhost] and IPs [16.32.12.161 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com localhost] and IPs [16.32.12.161 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0113 04:39:34.527846 4844 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0113 04:39:34.678520 4844 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0113 04:39:34.781723 4844 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0113 04:39:34.871489 4844 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0113 04:39:34.967682 4844 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0113 04:39:35.110542 4844 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0113 04:39:35.110632 4844 manifests.go:102] [control-plane] getting StaticPodSpecs
I0113 04:39:35.110757 4844 certs.go:519] validating certificate period for CA certificate
I0113 04:39:35.110810 4844 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0113 04:39:35.110836 4844 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0113 04:39:35.110856 4844 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I0113 04:39:35.110876 4844 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0113 04:39:35.110898 4844 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0113 04:39:35.110919 4844 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0113 04:39:35.111328 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0113 04:39:35.111378 4844 manifests.go:102] [control-plane] getting StaticPodSpecs
I0113 04:39:35.111487 4844 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0113 04:39:35.111512 4844 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0113 04:39:35.111532 4844 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I0113 04:39:35.111553 4844 manifests.go:128] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0113 04:39:35.111573 4844 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0113 04:39:35.111593 4844 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0113 04:39:35.111613 4844 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0113 04:39:35.111633 4844 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0113 04:39:35.112014 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0113 04:39:35.112064 4844 manifests.go:102] [control-plane] getting StaticPodSpecs
I0113 04:39:35.112162 4844 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0113 04:39:35.113257 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I0113 04:39:35.113276 4844 kubelet.go:67] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
I0113 04:39:35.435924 4844 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
I0113 04:39:35.436329 4844 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0113 04:39:45.445701 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10008 milliseconds
I0113 04:39:55.947643 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:40:06.447590 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
[kubelet-check] Initial timeout of 40s passed.
I0113 04:40:16.947274 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:40:27.449973 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:40:37.948672 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds
I0113 04:40:48.454894 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:40:58.947357 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:41:09.447811 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds
I0113 04:41:19.948021 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:41:24.015319 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 3569 milliseconds
I0113 04:41:34.448399 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:41:44.947303 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds
I0113 04:41:55.047823 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10100 milliseconds
I0113 04:42:05.048792 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:42:15.447846 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:42:25.953775 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10006 milliseconds
I0113 04:42:35.958271 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10004 milliseconds
I0113 04:42:45.044447 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 8592 milliseconds
I0113 04:42:55.454360 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10007 milliseconds
I0113 04:43:05.947112 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:43:06.466243 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 518 milliseconds
I0113 04:43:16.966248 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:43:27.446466 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:43:27.978857 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 532 milliseconds
I0113 04:43:38.447348 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds
I0113 04:43:48.948740 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10002 milliseconds
I0113 04:43:58.957933 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10008 milliseconds
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:108
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
cmd/kubeadm/app/cmd/init.go:111
github.com/spf13/cobra.(*Command).execute
vendor/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
cmd/kubeadm/app/cmd/phases/workflow/runner.go:260
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
cmd/kubeadm/app/cmd/init.go:111
github.com/spf13/cobra.(*Command).execute
vendor/github.com/spf13/cobra/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
vendor/github.com/spf13/cobra/command.go:1068
github.com/spf13/cobra.(*Command).Execute
vendor/github.com/spf13/cobra/command.go:992
k8s.io/kubernetes/cmd/kubeadm/app.Run
cmd/kubeadm/app/kubeadm.go:50
main.main
cmd/kubeadm/kubeadm.go:25
runtime.main
/usr/local/go/src/runtime/proc.go:250
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause
f09fd6b7be496 9ecc4287300e3 About a minute ago Exited kube-apiserver 4 7766ed4728e56 kube-apiserver-cvs-k8s-master1.hpemf.com
54895a4902925 babc03668f18a 4 minutes ago Running kube-scheduler 0 22bfa846906f3 kube-scheduler-cvs-k8s-master1.hpemf.com
d61e404e4c8ee c527ad14e0cd5 4 minutes ago Running kube-controller-manager 0 b4c9d707087be kube-controller-manager-cvs-k8s-master1.hpemf.com
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ crictl logs f09fd6b7be496
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the defaul t settings are now deprecated, you should set the endpoint instead.
E0113 04:44:30.250083 6325 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"" containerID="f09fd6b7be496"
FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ crictl logs f09fd6b7be496 -p
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the defaul t settings are now deprecated, you should set the endpoint instead.
E0113 04:44:48.366198 6381 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"" containerID="f09fd6b7be496"
FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID
E0113 04:45:21.658814 6504 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container "CONTAINERID": ot found" containerID="CONTAINERID"
FATA[0000] rpc error: code = NotFound desc = an error occurred when try to find container "CONTAINERID": not found
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ journalctl -xeu kubelet
Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:45:58.806025 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout
Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:45:58.806618 5121 trace.go:236] Trace[1316285868]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:150 (13-Jan-2024 04:45:28.805) (total time: 30001ms):
Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[1316285868]: ---"Objects listed" error:Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout 30000ms (04:45:58.806)
Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[1316285868]: [30.001489569s] [30.001489569s] END
Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:45:58.807015 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout
Jan 13 04:45:59 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:45:59.568418 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" interval="7s"
Jan 13 04:46:06 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:06.426353 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cvs-k8s-master1.hpemf.com" not found"
Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.015055 5121 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSecond>
Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:08.018895 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee"
Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.019047 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.019821 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b"
Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:10.018782 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375"
Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:10.018938 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:10.019463 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "etcd" with CrashLoopBackOff: "back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279"
Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794748 5121 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://16.32.12.161:6443/api/v1/nodes\": dial tcp 16.32.12.161:6443: connect: no route to host" node="cvs-k8s-master1.hpemf.com"
Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794757 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": dial tcp 16.32.12.161:6443: connect: no route to host" interval="7s"
Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794920 5121 certificate_manager.go:562] kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post "https://16.32.12.161:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 16.32.12.161:6443: connect: no route to host
Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:46:13.796329 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.RuntimeClass: Get "https://16.32.12.161:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: connect: no route to host
Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.796492 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://16.32.12.161:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: connect: no route to host
Jan 13 04:46:16 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:16.426807 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cvs-k8s-master1.hpemf.com" not found"
Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:19.021144 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee"
Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:19.022454 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:19.024282 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b"
Jan 13 04:46:20 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:20.802260 5121 kubelet_node_status.go:70] "Attempting to register node" node="cvs-k8s-master1.hpemf.com"
Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:23.019339 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375"
Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:23.019554 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:23.020038 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "etcd" with CrashLoopBackOff: "back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279"
Jan 13 04:46:26 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:26.427263 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cvs-k8s-master1.hpemf.com" not found"
Jan 13 04:46:30 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:30.800939 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": dial tcp 16.32.12.161:6443: i/o timeout" interval="7s"
Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:31.019007 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee"
Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.019153 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.019949 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b"
Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.021470 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:36 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:36.429360 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cvs-k8s-master1.hpemf.com" not found"
Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:37.019963 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375"
Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:37.020194 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:37.022300 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "etcd" with CrashLoopBackOff: "back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279"
Jan 13 04:46:39 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:39.019158 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:45.019159 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee"
Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:45.019289 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:45.021634 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b"
Jan 13 04:46:46 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:46.430534 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "cvs-k8s-master1.hpemf.com" not found"
Jan 13 04:46:47 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:47.803608 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" interval="7s"
Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.017029 5121 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSecond>
Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.017228 5121 event.go:228] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds>
Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:48.019835 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375"
Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.020044 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1"
Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.020590 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to "StartContainer" for "etcd" with CrashLoopBackOff: "back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279"
Jan 13 04:46:50 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:50.803379 5121 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "https://16.32.12.161:6443/api/v1/nodes\": dial tcp 16.32.12.161:6443: i/o timeout" node="cvs-k8s-master1.hpemf.com"
Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:46:55.047513 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout
Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:55.047672 5121 trace.go:236] Trace[581891402]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:150 (13-Jan-2024 04:46:25.046) (total time: 30000ms):
Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[581891402]: ---"Objects listed" error:Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout 30000ms (04:46:55.047)
Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[581891402]: [30.000786886s] [30.000786886s] END
Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:55.047699 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout
lines 2844-2897/2897 (END)
Has anyone face such problem
What version of containerd are you using?
containerd --version containerd containerd.io 1.6.26 3dd1e88
Any other relevant information
cvs-k8s-master1 $ crictl info
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
E0113 05:19:17.843289 7784 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory""
FATA[0000] getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $
cvs-k8s-master1 $ uname -a
Linux cvs-k8s-master1.hpemf.com 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
cvs-k8s-master1 $
Show configuration if it is related to CRI plugin.
mv /etc/containerd/config.toml /etc/containerd/config.toml_old
sudo systemctl restart containerd
sudo systemctl status containerd
cvs-k8s-master1 $ cat /etc/containerd/config.toml_old
Copyright 2018-2022 Docker Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
disabled_plugins = ["cri"]
#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
#[grpc]
address = "/run/containerd/containerd.sock"
uid = 0
gid = 0
#[debug]
address = "/run/containerd/debug.sock"
uid = 0
gid = 0
level = "info"
The text was updated successfully, but these errors were encountered: