kube-apiserver restart failed after edit /etc/kubernetes/manuals/kube-apiserver. yaml

[chaseSpace]

$ crictl ps -a
CONTAINER           IMAGE               CREATED                  STATE               NAME                      ATTEMPT             POD ID              POD
4183c2a4d8cc8       48f6f02f2e904       Less than a second ago   Exited              kube-apiserver            4                   d489ecf68d547       kube-apiserver-k8s-master
b23b9d7b3924a       2fdc9124e4ab3       Less than a second ago   Exited              kube-controller-manager   5                   9f1dc48a013aa       kube-controller-manager-k8s-master
70bd14d1409b1       62a4b43588914       Less than a second ago   Exited              kube-scheduler            5                   daadc53356cf8       kube-scheduler-k8s-master
49975b7b126f7       48f6f02f2e904       1 second ago             Running             kube-apiserver            5                   bd06da85d6ba8       kube-apiserver-k8s-master
ffeb275ad12a2       62a4b43588914       26 seconds ago           Running             kube-scheduler            6                   daadc53356cf8       kube-scheduler-k8s-master
54a1de5248a18       2fdc9124e4ab3       29 seconds ago           Running             kube-controller-manager   6                   9f1dc48a013aa       kube-controller-manager-k8s-master
3aec67de5a5a3       5185b96f0becf       43 hours ago             Running             coredns                   0                   c12bce5969e62       coredns-c676cc86f-8vpk4
4dd8d277c01e3       5185b96f0becf       43 hours ago             Running             coredns                   0                   4ac7a129fe58d       coredns-c676cc86f-fzzjp
eb790adc7717c       01cdfa8dd262f       43 hours ago             Running             kube-flannel              0                   bdee08d7d1067       kube-flannel-ds-v4hzd
81a0ff3b143f1       01cdfa8dd262f       43 hours ago             Exited              install-cni               0                   bdee08d7d1067       kube-flannel-ds-v4hzd
05048cc96fbd7       a55d1bad692b7       43 hours ago             Exited              install-cni-plugin        0                   bdee08d7d1067       kube-flannel-ds-v4hzd
ba096938c2bd6       b2d7e01cd611a       43 hours ago             Running             kube-proxy                0                   699b2989bd346       kube-proxy-t64w8
5f1532aa85aff       fce326961ae2d       43 hours ago             Running             etcd                      2                   9638f3b3d77ed       etcd-k8s-master
$ crictl logs -f 4183c2a4d8cc8                    
FATA[0000] failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-apiserver-k8s-master_7b0f266280b6524ac189d7737f6df363/kube-apiserver/4.log": lstat /var/log/pods/kube-system_kube-apiserver-k8s-master_7b0f266280b6524ac189d7737f6df363/kube-apiserver/4.log: no such file or directory 

I encountered this issue after editing '/etc/kubernetes/manuals/kube-apiserver. yaml'. Even if the file is restored, it still cannot run.

I dont know what can i do now...

[chaseSpace]

I removed kube-apiserver pod with command crictl rm 4183c2a4d8cc8, and after about ten seconds, a new pod automatically appeared again, but it still Exited. Below is log:

$ crictl logs -f 5956ba6ab590e            
I1118 20:14:41.352690       1 server.go:565] external host was not specified, using 192.168.31.2
I1118 20:14:41.353208       1 server.go:162] Version: v1.25.14
I1118 20:14:41.353250       1 server.go:164] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1118 20:14:41.808489       1 shared_informer.go:252] Waiting for caches to sync for node_authorizer
I1118 20:14:41.808723       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1118 20:14:41.808734       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.

omit...

W1118 20:14:42.363344       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W1118 20:14:42.363372       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W1118 20:14:42.364468       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W1118 20:14:42.364495       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W1118 20:14:42.364529       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W1118 20:14:42.367064       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W1118 20:14:42.367087       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W1118 20:14:42.380050       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.

omit...

I1118 20:14:43.306206       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I1118 20:14:43.306402       1 secure_serving.go:210] Serving securely on [::]:6443
I1118 20:14:43.306654       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I1118 20:14:43.320317       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1118 20:14:43.320369       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I1118 20:14:43.321063       1 controller.go:83] Starting OpenAPI AggregationController
I1118 20:14:43.321217       1 controller.go:80] Starting OpenAPI V3 AggregationController
I1118 20:14:43.321734       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I1118 20:14:43.322098       1 available_controller.go:491] Starting AvailableConditionController
I1118 20:14:43.322103       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I1118 20:14:43.322181       1 aggregator.go:151] waiting for initial CRD sync...
I1118 20:14:43.322355       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I1118 20:14:43.322388       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I1118 20:14:43.322394       1 shared_informer.go:252] Waiting for caches to sync for cluster_authentication_trust_controller
I1118 20:14:43.322445       1 apf_controller.go:312] Starting API Priority and Fairness config controller
I1118 20:14:43.322738       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I1118 20:14:43.322743       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I1118 20:14:43.325250       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I1118 20:14:43.325256       1 shared_informer.go:252] Waiting for caches to sync for crd-autoregister
I1118 20:14:43.325279       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I1118 20:14:43.330709       1 controller.go:85] Starting OpenAPI controller
I1118 20:14:43.330749       1 controller.go:85] Starting OpenAPI V3 controller
I1118 20:14:43.330770       1 naming_controller.go:291] Starting NamingConditionController
I1118 20:14:43.330789       1 establishing_controller.go:76] Starting EstablishingController
I1118 20:14:43.330830       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I1118 20:14:43.330847       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I1118 20:14:43.330859       1 crd_finalizer.go:266] Starting CRDFinalizer
I1118 20:14:43.332130       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I1118 20:14:43.396812       1 controller.go:211] Shutting down kubernetes service endpoint reconciler

# first error here
E1118 20:14:43.397227       1 controller.go:214] Unable to remove endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.31.2, ResourceVersion: 0, AdditionalErrorMsg: 
I1118 20:14:43.397290       1 controller.go:89] Shutting down OpenAPI AggregationController
E1118 20:14:43.397653       1 shared_informer.go:255] unable to sync caches for crd-autoregister
I1118 20:14:43.397673       1 crdregistration_controller.go:116] Shutting down crd-autoregister controller

omit...

I1118 20:14:43.432486       1 dynamic_serving_content.go:146] "Shutting down controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I1118 20:14:43.411692       1 shared_informer.go:259] Caches are synced for node_authorizer
E1118 20:14:43.411707       1 customresource_discovery_controller.go:212] timed out waiting for caches to sync
E1118 20:14:43.418732       1 cache.go:35] Unable to sync caches for AvailableConditionController controller
I1118 20:14:43.425165       1 shared_informer.go:259] Caches are synced for cluster_authentication_trust_controller
I1118 20:14:43.432529       1 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"

[chaseSpace]

Then I continue to tried:

• kubeadm reset -f
• kubeadm init (single node)
• edit /etc/kubernetes/manuals/kube-apiserver. yaml

edit part：

spec:
  containers:
  - command:
    - kube-apiserver
    - --token-auth-file=/etc/kubernetes/k8s_account_tokens.csv   <------------- new line added
    - --advertise-address=192.168.31.2

But kube-apiserver pod report no such file... for token-auth-file added：

➜  practice crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD
4389df402233a       48f6f02f2e904       2 seconds ago       Exited              kube-apiserver            4                   2d7f04230740f       kube-apiserver-k8s-master
2f3fcac27979d       62a4b43588914       6 minutes ago       Running             kube-scheduler            1                   1a9bfdeacb631       kube-scheduler-k8s-master
4261cad5bc6f3       2fdc9124e4ab3       6 minutes ago       Running             kube-controller-manager   1                   95b15dd3eec53       kube-controller-manager-k8s-master
6387487eb96a3       b2d7e01cd611a       6 minutes ago       Running             kube-proxy                0                   c4537525bb235       kube-proxy-k5wst
b149955b883da       62a4b43588914       7 minutes ago       Exited              kube-scheduler            0                   1a9bfdeacb631       kube-scheduler-k8s-master
af95214f6a93b       2fdc9124e4ab3       7 minutes ago       Exited              kube-controller-manager   0                   95b15dd3eec53       kube-controller-manager-k8s-master
9ec75b0d0ecbe       fce326961ae2d       7 minutes ago       Running             etcd                      0                   822a8fef1c695       etcd-k8s-master

➜  practice crictl logs 4389df402233a            
I1118 20:46:34.221643       1 server.go:565] external host was not specified, using 192.168.31.2
I1118 20:46:34.222252       1 server.go:162] Version: v1.25.14
I1118 20:46:34.222290       1 server.go:164] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E1118 20:46:34.650376       1 run.go:74] "command failed" err="open /etc/kubernetes/k8s_account_tokens.csv: no such file or directory"

# but it exactly exist!
➜  practice cat /etc/kubernetes/k8s_account_tokens.csv                                        
nlZtQeHoS8k0Pvbe,user3,3
nxdt123445k0P21d,user4,4
➜  practice ls -l /etc/kubernetes/k8s_account_tokens.csv 
-rwxrwxrwx. 1 root root 50 Nov 19 03:06 /etc/kubernetes/k8s_account_tokens.csv

[chaseSpace]

I have opened an issue on kubernetes/kubernetes#121961