[Bug] Kyverno image verification does not work on pre existing resources

[andriktr]

Kyverno Version

1.11.4

Description

Hey,
I’m trying to confgure image-verification process for my environment. Here is my policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: if-baltic-verify-image-signature
  annotations:
    policies.kyverno.io/title: Verify Image
    policies.kyverno.io/category: Software Supply Chain Security
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/minversion: 1.7.0
    policies.kyverno.io/description: >-
      Using the Cosign project, OCI images may be signed to ensure supply chain
      security is maintained. Those signatures can be verified before pulling into
      a cluster. This policy checks the signature of an image repo called
      ghcr.io/kyverno/test-verify-image to ensure it has been signed by verifying
      its signature against the provided public key. This policy serves as an illustration for
      how to configure a similar rule and will require replacing with your image(s) and keys. 
spec:
  validationFailureAction: Audit
  background: true
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-image-signature
      match:
        any:
        - resources:
            kinds:
            - Pod      
            namespaces:
            - consul-experimental-apps           
      verifyImages:
      - imageReferences:
        - '*'
        mutateDigest: false
        verifyDigest: false
        attestors:
        - count: 1
          entries:
          - keys:
              ctlog:
                ignoreSCT: true
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                edited
                -----END PUBLIC KEY-----
              rekor:
                ignoreTlog: true
                url: https://rekor.sigstore.dev/

The policy it’s self seems to be working fine but validation is failing with policy if-baltic-verify-image-signature/verify-image-signature fail: image is not verified for unspecified reason. In kyverno logs I see the following:

kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.714633       1 event_broadcaster.go:338] "Event occurred" object="if-baltic-verify-image-signature" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" action="Resource Passed" note="Pod consul-experimental-apps/consul-server-74f6ff7dcd-vzg5h: [verify-image-signature] fail; image is not verified"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.724744       1 event_broadcaster.go:338] "Event occurred" object="consul-experimental-apps/consul-server-74f6ff7dcd-vzg5h" kind="Pod" apiVersion="v1" type="Warning" reason="PolicyViolation" action="Resource Passed" note="policy if-baltic-verify-image-signature/verify-image-signature fail: image is not verified"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.887534       1 image.go:76] "missing image metadata in annotation" logger="engine.validate" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.907251       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.907291       1 imageverifier.go:261] "cache entry found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-k8s-control-plane:1.4.1"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.908000       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.908029       1 imageverifier.go:261] "cache entry found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-dataplane:1.4.1"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.908662       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.908688       1 imageverifier.go:265] "cache entry not found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-green:0.1.0"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.908705       1 imageverifier.go:321] "verifying image signatures" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" image="privateazureregistry.azurecr.io/consul-green:0.1.0" attestors=1 attestations=0
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.910343       1 auth.go:129] "logging in to Azure ACR for privateazureregistry.azurecr.io/consul-green"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:36.976996       1 image.go:76] "missing image metadata in annotation" logger="engine.validate" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.000140       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.000194       1 imageverifier.go:261] "cache entry found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-k8s-control-plane:1.4.1"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.001584       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.001613       1 imageverifier.go:261] "cache entry found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-dataplane:1.4.1"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.002531       1 imageverifier.go:115] "missing image metadata in annotation" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" key="kyverno.io/verify-images"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.002577       1 imageverifier.go:265] "cache entry not found" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" namespace="" policy="if-baltic-verify-image-signature" ruleName="verify-image-signature" imageRef="privateazureregistry.azurecr.io/consul-green:0.1.0"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.002594       1 imageverifier.go:321] "verifying image signatures" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-zvtsq" rule.name="verify-image-signature" image="privateazureregistry.azurecr.io/consul-green:0.1.0" attestors=1 attestations=0
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.004184       1 auth.go:129] "logging in to Azure ACR for privateazureregistry.azurecr.io/consul-green"
kyverno-reports-controller-689fb56c49-bwfh2:controller I0514 11:57:37.207262       1 imageverifier.go:489] "image attestors verification succeeded" logger="engine.verify" policy.name="if-baltic-verify-image-signature" policy.namespace="" policy.apply="All" new.kind="Pod" new.namespace="consul-experimental-apps" new.name="green-78648b9cfc-fj5gp" rule.name="verify-image-signature" verifiedCount=1 requiredCount=1

Images are stored in Azure Container Registry. Both kyverno reports and admission controller are congiured to use workload identity and are able to reach ACR with pull permissions.
If I try to verify signature manualy i.e. with
cosign verify --key cosign-dev.pub --private-infrastructure privateazureregistry.azurecr.io/consul-k8s-control-plane:1.4.1
images are verified successfully.

Also if I put policy in enforce mode it works correctly and signed images are allowed and not signed are blocked. However reporting still shows same verify-image-signature fail: image is not verified
Any thoughts?
Thank you.

Slack discussion

https://kubernetes.slack.com/archives/CLGR9BJU9/p1715688426417939

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.