[Bug] kyverno-admission-controller excessively updating the manifest for kyverno-resource-validating-webhook-cfg

[flickers]

Kyverno Version

1.12.0

Kubernetes Version

1.28.x

Kubernetes Platform

EKS

Kyverno Rule Type

Validate

Description

Happens in both EKS and AKS
I'm running the latest version of Kyverno (ghcr.io/kyverno/kyverno:v1.12.1) and Kyverno-policies
I'm using the kyverno helm chart (3.2.2) and ArgoCD (latest version)

The kyverno-admission-controller seems to be updating validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg very frequently (every 2 seconds or so according to audit logs in EKS)
Can be seen by running kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io kyverno-resource-validating-webhook-cfg -o yaml -w
It seems to be flapping the rule.operations for resources pods, pods/ephemeralcontainers and replicationcontrollers

rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:      # Notice this section here
    - CREATE
    - UPDATE
    - DELETE
    - CONNECT
    resources:
    - pods
    - pods/ephemeralcontainers
    - replicationcontrollers
    scope: Namespaced

rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:      # Notice this section here
    - DELETE
    - CONNECT
    - CREATE
    - UPDATE
    resources:
    - pods
    - pods/ephemeralcontainers
    - replicationcontrollers
    scope: Namespaced

This just keeps flapping back and forth as can be seen in the number of generations

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io kyverno-resource-validating-webhook-cfg -o=custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation
NAME                                      GENERATION
kyverno-resource-validating-webhook-cfg   127286

Steps to reproduce

1. I assume that you should be able to reproduce by deploying kyverno helm chart and kyverno-policies helm chart (not sure that this has anything to do with kyverno-policies though)

Expected behavior

kyverno-admission-controller should not be updating the manifest for kyverno-resource-validating-webhook-cfg every other second

Screenshots

No response

Kyverno logs

Not seeing anything worth posting here from the logs

Slack discussion

https://kubernetes.slack.com/archives/CLGR9BJU9/p1715693532811989

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[realshuting]

This issue should be fixed in 1.12.1, can you help verify?

[flickers]

@realshuting well, we are already running version 1.12.1 and are still seeing this.

[wsdnv]

We also experience this in v1.12.1 on Azure, Helm chart 3.2.2. In our case additionally it tends to make Admission Controller to be stuck with the message below at some point, especially if the Kubernetes API slowed down for a moment.

2024-05-20T08:19:13Z	DEBUG	webhook-controller.worker	controller/run.go:115	Retrying request	{"id": 0, "obj": "kyverno-resource-validating-webhook-cfg", "error": "Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"kyverno-resource-validating-webhook-cfg\": the object has been modified; please apply your changes to the latest version and try again"}
2024-05-20T08:19:15Z	DEBUG	webhook-controller.worker	controller/run.go:115	Retrying request	{"id": 0, "obj": "kyverno-resource-validating-webhook-cfg", "error": "Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"kyverno-resource-validating-webhook-cfg\": the object has been modified; please apply your changes to the latest version and try again"}
2024-05-20T08:19:22Z	DEBUG	webhook-controller.worker	controller/run.go:115	Retrying request	{"id": 0, "obj": "kyverno-resource-validating-webhook-cfg", "error": "Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"kyverno-resource-validating-webhook-cfg\": the object has been modified; please apply your changes to the latest version and try again"}

[flickers]

I'm using the kyverno-policies helm chart version 3.2.1
I think we are just using the defaults in the values except podSecurityStandard = restricted

kyverno-policies:
  # -- Pod Security Standard profile (`baseline`, `restricted`, `privileged`, `custom`).
  # For more info https://kyverno.io/policies/pod-security.
  podSecurityStandard: restricted

  # -- Pod Security Standard (`low`, `medium`, `high`).
  podSecuritySeverity: medium

  # -- Policies to include when `podSecurityStandard` is `custom`.
  podSecurityPolicies: []

  # -- Additional policies to include from `other`.
  includeOtherPolicies: []
  # - require-non-root-groups

  # -- Additional policies to include from `restricted`.
  includeRestrictedPolicies: []
  # - require-run-as-non-root-user

  # -- API server behavior if the webhook fails to respond ('Ignore', 'Fail')
  # For more info: https://kyverno.io/docs/writing-policies/policy-settings/
  failurePolicy: Ignore

  # -- Validation failure action (`audit`, `enforce`).
  # For more info https://kyverno.io/docs/writing-policies/validate.
  validationFailureAction: audit

  # -- Define validationFailureActionByPolicy for specific policies.
  # Override the defined `validationFailureAction` with a individual validationFailureAction for individual Policies.
  validationFailureActionByPolicy: {}
  #  disallow-capabilities-strict: enforce
  #  disallow-host-path: enforce
  #  disallow-host-ports: enforce

  # -- Define validationFailureActionOverrides for specific policies.
  # The overrides for `all` will apply to all policies.
  validationFailureActionOverrides:
    all: []
    # all:
    #   - action: audit
    #     namespaces:
    #       - ingress-nginx
    # disallow-host-path:
    #   - action: audit
    #     namespaces:
    #       - fluent

  # -- Exclude resources from individual policies.
  # Policies with multiple rules can have individual rules excluded by using the name of the rule as the key in the `policyExclude` map.
  policyExclude: {}
    # # Exclude resources from individual policies
    # disallow-host-path:
    #   any:
    #   - resources:
    #       kinds:
    #       - Pod
    #       namespaces:
    #       - fluent
    # # Policies with multiple rules can have individual rules excluded
    # adding-capabilities-strict:
    #   any:
    #   - resources:
    #       kinds:
    #       - Pod
    #       namespaces:
    #       - kube-system
  # -- Add preconditions to individual policies.
  # Policies with multiple rules can have individual rules excluded by using the name of the rule as the key in the `policyPreconditions` map.
  policyPreconditions: {}
    # # Exclude resources from individual policies
    # require-run-as-non-root-user:
    #   all:
    #   - key: "{{ request.object.metadata.name }}"
    #     operator: NotEquals
    #     value: "dcgm-exporter*"
    # # Policies with multiple rules can have individual rules excluded
    # require-drop-all:
    #   any:
    #   - key: "{{ request.object.metadata.name }}"
    #     operator: NotEquals
    #     value: "dcgm-exporter*"
    # adding-capabilities-strict:
    #   all:
    #   - key: "{{ request.object.metadata.name }}"
    #     operator: NotEquals
    #     value: "dcgm-exporter*"

  # -- Name override.
  nameOverride:

  # -- Additional labels.
  customLabels: {}

  # -- Policies background mode
  background: false

  # -- Kyverno version
  # The default of "autodetect" will try to determine the currently installed version from the deployment
  kyvernoVersion: autodetect

[realshuting]

Thanks @flickers - I'm able to reproduce the issue, and sent the fixing PR #10274.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: wildcard-support-in-matchlabels
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  admission: true
  background: false
  rules:
  - match:
      any:
      - resources:
          kinds:
          - ReplicationController
    name: wildcard-label-rc
    validate:
      message: Using a mutable image tag e.g. 'latest' is not allowed.
      pattern:
        spec:
          template:
            spec:
              containers:
              - image: '!*:latest'
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: wildcard-label
    validate:
      message: Using a mutable image tag e.g. 'latest' is not allowed.
      pattern:
        spec:
          containers:
          - image: '!*:latest'