Hanging Cilium Operator

[toschneck]

What happened?

On some changes of the cilium values of the system application, in my case added the following ingress config, the deployment of the cilium operator hangs and the helm deployment don't get finished. Somehow a k rollout restart deployment cilium-operator fixing the problem. But as is quite non-transparent for users, we should fix the behavour.

### added values
ingressController:
  enabled: true
  loadbalancerMode: shared
  default: true
  enforceHttps: false
envoy:
  enabled: true

Possible Solution
If we may can add some label to the operator deployment, what ensures that the operator get restarted, e.g. last-update:date.

Expected behavior

Cilium config change get rolled out properly.

How to reproduce the issue?

Create a Cilium CNI based cluster, edit the values of the cilium system application and add the above yaml.

check kube-system namespace hanging pods, after the change

kube-system            cilium-7ljnp                                    0/1     Running   0               69s
kube-system            cilium-envoy-fhxzq                              1/1     Running   0               70s
kube-system            cilium-envoy-l7fnk                              1/1     Running   0               70s
kube-system            cilium-operator-79fdc88454-fw58x                1/1     Running   0               8m14s
kube-system            cilium-pxrb4                                    0/1     Running   0               69s

Restart operator, fixes the problem.

How is your environment configured?

• KKP version: 2.24.0
• Shared or separate master/seed clusters?: shared

Provide your KKP manifest here (if applicable)

https://github.com/kubermatic/sig-cs-infra/tree/main/lab.kubermatic.io

What cloud provider are you running on?

tested on AWS

What operating system are you running in your user cluster?

Ubuntu

Additional information

[xrstf]

When I try to reproduce this, adding the YAML snippet you provided yields an error:

admission webhook "applicationinstallations.apps.kubermatic.k8c.io" denied the request: ApplicationInstallation validation request 55773472-939f-4071-b8e2-462febf01b0e denied: [spec.values.cni: Invalid value: "null": value is immutable spec.values.ipam: Invalid value: "null": value is immutable spec.values.kubeProxyReplacement: Not found: "null" spec.values.operator: Invalid value: "null": value missing or incorrect spec.values.operator.securityContext: Invalid value: "null": value is immutable]

Am using Cilium 1.14.3.

[cnvergence]

found the same issue, seems like the dashboard is sending only updated valuesBlock, while values are empty, causing validation failure