Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] No the error message/note field in the Kyverno logs when using Validate Rules (version 1.12) #10170

Open
2 tasks done
TheKnowledgeKeeper opened this issue May 3, 2024 · 2 comments
Assignees
Labels
bug Something isn't working log enhancement validation Issues pertaining to the validate ability.

Comments

@TheKnowledgeKeeper
Copy link

TheKnowledgeKeeper commented May 3, 2024

Kyverno Version

1.12

Kubernetes Version

1.24

Kubernetes Platform

GKE

Description

In Kyverno version 1.12, I don't see any msg or note fields in Kyverno logs that need to be displayed when using the validate rule type. But in version 1.11 it is fully available.

My policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: slsa-verify-all-images
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  background: true
  rules:
  - name: slsa-restrict-third-party-images
    match:
      any:
      - resources:
          kinds:
          - Pod
          operations:
          - CREATE
          - UPDATE
    exclude:
      any:
      - resources:
          kinds:
            - ReplicaSet
      - resources:
          namespaces:
            - kyverno
            - kube-system
    skipBackgroundRequests: true
    validate:
      message: "Your third-party images have not been whitelisted {{ request.object.spec.[ephemeralContainers, initContainers, containers][].image }}"
      foreach:
      - list: "request.object.spec.[ephemeralContainers, initContainers, containers][]"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
        deny:
          conditions:
            all:
              - key: '{{ imageData.registry || "" }}'
                operator: NotEquals
                value: gcr.io
              - key: '{{ imageData.resolvedImage || "" }}'
                operator: AnyNotIn
                value:
                - index.docker.io/apache/airflow@sha256:*
                - index.docker.io/apache/superset@sha256:*
                - index.docker.io/bitnami/git@sha256:*
                - index.docker.io/bitnami/postgresql@sha256:*
                - index.docker.io/bitnami/redis@sha256:*
                - index.docker.io/curlimages/curl@sha256:*

Manifest of my pod applied

apiVersion: v1
kind: Pod
metadata:
  name: taile-test-kyverno-pod
  namespace: default
  labels:
    app: taile-test-kyverno-pod
spec:
  containers:
  - name: taile
    image: python:alpine3.19
    tty: true

The error message I get from the console screen

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/default/taile-test-kyverno-pod was blocked due to the following policies

slsa-verify-all-images:
  slsa-restrict-third-party-images: 'validation failure: Your third-party images have
    not been whitelisted ["python:alpine3.19"]'

Below are the types of logs I get for each version

Version 1.11 - This log has the note field describing the error message

{
    "@timestamp": 1714365065.994449,
    "k8s_container_hash": "ghcr.io/kyverno/kyverno@sha256:76d0252892bab3b7682444f283660274977a6243450720a3893221b510eed94e",
    "container_name": "/k8s_kyverno_kyverno-admission-controller-8694857c99-zfw9t_kyverno",
    "k8s_container_name": "kyverno",
    "k8s_namespace_name": "kyverno",
    "k8s_docker_id": "8100b91c1586326d7180edae8ef298d3c2050dea67d5131cc2247e42ce53d905",
    "k8s_workload_path": "staging.kyverno.kyverno-admission-controller",
    "_p": "F",
    "k8s_container_image": "ghcr.io/kyverno/kyverno:v1.11.4",
    "stream": "stderr",
    "time": "2024-04-29T04:31:05.994449375Z",
    "k8s_pod_name": "kyverno-admission-controller-8694857c99-zfw9t",
    "fluentd_event_source": "flb-kafka-26mzb",
    "log":
    {
        "level": "info",
        "ts": 1714365065.9941533,
        "logger": "klog",
        "caller": "events/event_broadcaster.go:338",
        "msg": "Event occurred",
        "object":
        {
            "name": "imageref-demo",
            "namespace": "watchtower-v"
        },
        "kind": "Policy",
        "apiVersion": "kyverno.io/v1",
        "type": "Warning",
        "reason": "PolicyViolation",
        "action": "Resource Blocked",
        "note": "Pod watchtower-v/taile-test-kyverno-pod: [no-root-images] fail (blocked); validation failure: Your third-party images have not been whitelisted [\"python:alpine3.19\"]"
    },
    "k8s_pod_id": "a869eead-58c6-417b-be2f-6fa9611b1407",
    "k8s_cluster_name": "staging",
    "k8s_host": "gke-k8s-prod-staging-np-e2-4-8-2d44b41e-snjw",
    "k8s_labels":
    {
        "app.kubernetes.io/instance": "kyverno",
        "app.kubernetes.io/managed-by": "Helm",
        "app.kubernetes.io/part-of": "kyverno",
        "app.kubernetes.io/version": "3.1.4",
        "pod-template-hash": "8694857c99",
        "app.kubernetes.io/component": "admission-controller",
        "helm.sh/chart": "kyverno-3.1.4"
    },
    "container_img": "ghcr.io/kyverno/kyverno@sha256:76d0252892bab3b7682444f283660274977a6243450720a3893221b510eed94e"
}

Version 1.12 - I get 4 logs but no field describing the error message: Your third-party images have not been whitelisted ...

{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:35Z",
        "logger": "webhooks.resource.validate",
        "caller": "validation/validation.go:116",
        "msg": "validation failed",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "Enforce",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images",
        "failed rules":
        [
            "slsa-restrict-third-party-images"
        ]
    }
{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "validation/validation.go:116",
        "msg": "validation failed",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "Enforce",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images",
        "failed rules":
        [
            "slsa-restrict-third-party-images"
        ]
    }
{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "utils/block.go:29",
        "msg": "blocking admission request",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "validate",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images"
    }
{
        "level": "info",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "resource/handlers.go:146",
        "msg": "admission request denied",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": ""
    }

Steps to reproduce

  1. Upgrade kyverno version to v1.12 .
  2. Apply My policy in above description .
  3. Use kubectl to capture logs kubectl logs -f -l app.kubernetes.io/part-of=kyverno -n kyverno | grep "taile-test-kyverno-pod" (taile-test-kyverno-pod is my pod name) .
  4. Apply Manifest of my pod .
  5. Check logs get from step 3 .

Expected behavior

I would like the log information to be fully described as in v1.11, which I showed above.

"log":
    {
        "level": "info",
        "ts": 1714365065.9941533,
        "logger": "klog",
        "caller": "events/event_broadcaster.go:338",
        "msg": "Event occurred",
        "object":
        {
            "name": "imageref-demo",
            "namespace": "watchtower-v"
        },
        "kind": "Policy",
        "apiVersion": "kyverno.io/v1",
        "type": "Warning",
        "reason": "PolicyViolation",
        "action": "Resource Blocked",
        "note": "Pod watchtower-v/taile-test-kyverno-pod: [no-root-images] fail (blocked); validation failure: Your third-party images have not been whitelisted [\"python:alpine3.19\"]"
    },

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

  • I have read and followed the documentation AND the troubleshooting guide.
  • I have searched other issues in this repository and mine is not recorded.
@TheKnowledgeKeeper TheKnowledgeKeeper added the bug Something isn't working label May 3, 2024
@chipzoller chipzoller transferred this issue from kyverno/policies May 3, 2024
@chipzoller chipzoller added log enhancement validation Issues pertaining to the validate ability. labels May 3, 2024
@chipzoller
Copy link
Member

Not a policy issue but code issue. Transferred to kyverno/kyverno.

@Manoramsharma
Copy link

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working log enhancement validation Issues pertaining to the validate ability.
Projects
None yet
Development

No branches or pull requests

3 participants