Watch() returns wrong "DELETED" event

[felixkrohn]

What happened (please include outputs or screenshots):

A python script watching the existence of a set of configmaps gets notified that a CM got deleted when this is not the case.
Upon investigation, we found that this event is triggered when a CM does not match the label selector anymore (i.e. the selected label is removed).

The fact that the CM is removed from the list of watched CMs does not imply it has been deleted, meaning this classifies as a bug,

What you expected to happen:

• The CM that was formerly watched, and whose labels changed, should just not be watched anymore.
• the python client should mimick the behaviour of kubectl (no event is sent)

$ kubectl  wait configmap --for=delete -l foo=bar &
[1] 240648
$ kubectl label configmap foo foo-
rolebinding.rbac.authorization.k8s.io/foo unlabeled
$ fg
kubectl wait configmap --for=delete -l foo=bar
error: timed out waiting for the condition on rolebindings/foo

How to reproduce it (as minimally and precisely as possible):

• run the script (change namespace and labelselector accordingly)
• send it to background or open another shell
• remove the watched label from a watched resource
• notice the message: ConfigMap 'foo' was deleted

#!/usr/bin/env python3
from kubernetes import client, config, watch
import os

config.load_kube_config()

v1 = client.CoreV1Api()
label_selector = "foo=bar"
namespace = "foo-test"

while True:
    try:
        stream = watch.Watch().stream(v1.list_namespaced_config_map, namespace=namespace, label_selector=label_selector)
        for event in stream:
            config_map = event['object']
            event_type = event['type']

            if event_type == 'DELETED':
                print(f"ConfigMap '{config_map.metadata.name}' was deleted")

    except Exception as e:
        print(f"Exception occurred: {str(e)}")
        continue

Anything else we need to know?:

Environment:

• Python version (python --version): 3.11.5
• Python client version (pip list | grep kubernetes): 24.2.0

[roycaihw]

Could you turn on DEBUG logging to see if the watch event is sent by the server? Reading the watch implementation I didn't see the client generating additional events that aren't sent by the server

[felixkrohn]

As I don't have access to the API server audit logs, I'm not sure how to properly debug what the server is sending.
But printing the whole event instead of only the 'type' gives me the following:

Event:
{'type': 'ADDED', 'object': {'api_version': 'v1',
 'binary_data': None,
 'data': {'foo': 'bar'},
 'immutable': None,
 'kind': 'ConfigMap',
 'metadata': {'annotations': None,
              'creation_timestamp': datetime.datetime(2024, 5, 8, 11, 29, 48, tzinfo=tzutc()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'labels': {'foo': 'bar'},
              'managed_fields': [{'api_version': 'v1',
                                  'fields_type': 'FieldsV1',
                                  'fields_v1': {'f:data': {'.': {},
                                                           'f:foo': {}}},
                                  'manager': 'kubectl-create',
                                  'operation': 'Update',
                                  'subresource': None,
                                  'time': datetime.datetime(2024, 5, 8, 11, 29, 48, tzinfo=tzutc())},
                                 {'api_version': 'v1',
                                  'fields_type': 'FieldsV1',
                                  'fields_v1': {'f:metadata': {'f:labels': {'.': {},
                                                                            'f:foo': {}}}},
                                  'manager': 'kubectl-label',
                                  'operation': 'Update',
                                  'subresource': None,
                                  'time': datetime.datetime(2024, 5, 13, 11, 31, 14, tzinfo=tzutc())}],
              'name': 'foo',
              'namespace': 'foo-test',
              'owner_references': None,
              'resource_version': '1010917341',
              'self_link': None,
              'uid': '6ea09c86-b6bc-40ff-8508-26dfa82c8c88'}},
'raw_object': {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'foo', 'namespace': 'foo-test', 'uid': '6ea09c86-b6bc-40ff-8508-26dfa82c8c88', 'resourceVersion': '1010917341', 'creationTimestamp': '2024-05-08T11:29:48Z', 'labels': {'foo': 'bar'}, 'managedFields': [{'manager': 'kubectl-create', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2024-05-08T11:29:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:foo': {}}}}, {'manager': 'kubectl-label', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2024-05-13T11:31:14Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:foo': {}}}}}]}, 'data': {'foo': 'bar'}}}

Event:
{'type': 'DELETED', 'object': {'api_version': 'v1',
 'binary_data': None,
 'data': {'foo': 'bar'},
 'immutable': None,
 'kind': 'ConfigMap',
 'metadata': {'annotations': None,
              'creation_timestamp': datetime.datetime(2024, 5, 8, 11, 29, 48, tzinfo=tzutc()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'labels': {'foo': 'bar'},
              'managed_fields': [{'api_version': 'v1',
                                  'fields_type': 'FieldsV1',
                                  'fields_v1': {'f:data': {'.': {},
                                                           'f:foo': {}}},
                                  'manager': 'kubectl-create',
                                  'operation': 'Update',
                                  'subresource': None,
                                  'time': datetime.datetime(2024, 5, 8, 11, 29, 48, tzinfo=tzutc())},
                                 {'api_version': 'v1',
                                  'fields_type': 'FieldsV1',
                                  'fields_v1': {'f:metadata': {'f:labels': {'.': {},
                                                                            'f:foo': {}}}},
                                  'manager': 'kubectl-label',
                                  'operation': 'Update',
                                  'subresource': None,
                                  'time': datetime.datetime(2024, 5, 13, 11, 31, 14, tzinfo=tzutc())}],
              'name': 'foo',
              'namespace': 'foo-test',
              'owner_references': None,
              'resource_version': '1010917412',
              'self_link': None,
              'uid': '6ea09c86-b6bc-40ff-8508-26dfa82c8c88'}},
'raw_object': {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'foo', 'namespace': 'foo-test', 'uid': '6ea09c86-b6bc-40ff-8508-26dfa82c8c88', 'resourceVersion': '1010917412', 'creationTimestamp': '2024-05-08T11:29:48Z', 'labels': {'foo': 'bar'}, 'managedFields': [{'manager': 'kubectl-create', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2024-05-08T11:29:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:foo': {}}}}, {'manager': 'kubectl-label', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2024-05-13T11:31:14Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:foo': {}}}}}]}, 'data': {'foo': 'bar'}}}

In the raw_object at the end of each event you can see 'operation': 'Update'.
Also as the CM is still there and has no deletionTimestamp set I can 100% affirm the API-server is not considering it DELETED.

Is the above output sufficient to dig further, or is there an easy way to debug the Watch function further?

Best regards,
Felix