Cannot list endpointslices using python kubernetes client

[bpinske]

What happened (please include outputs or screenshots):

Cannot list endpoint slices using the kubernetes python library in a cluster where one of the endpoint slices contains a null list of endpoints.

Fails with the following stack trace.

Traceback (most recent call last):
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/test.py", line 11, in <module>
    endpoint_slices = discovery_api.list_endpoint_slice_for_all_namespaces()
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api/discovery_v1_api.py", line 651, in list_endpoint_slice_for_all_namespaces
    return self.list_endpoint_slice_for_all_namespaces_with_http_info(**kwargs)  # noqa: E501
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api/discovery_v1_api.py", line 758, in list_endpoint_slice_for_all_namespaces_with_http_info
    return self.api_client.call_api(
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 192, in __call_api
    return_data = self.deserialize(response_data, response_type)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 264, in deserialize
    return self.__deserialize(data, response_type)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 303, in __deserialize
    return self.__deserialize_model(data, klass)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 639, in __deserialize_model
    kwargs[attr] = self.__deserialize(value, attr_type)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 280, in __deserialize
    return [self.__deserialize(sub_data, sub_kls)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 280, in <listcomp>
    return [self.__deserialize(sub_data, sub_kls)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 303, in __deserialize
    return self.__deserialize_model(data, klass)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 641, in __deserialize_model
    instance = klass(**kwargs)
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/models/v1_endpoint_slice.py", line 70, in __init__
    self.endpoints = endpoints
  File "/Users/bpinske/PycharmProjects/diff_k8_mass_inventory/venv/lib/python3.9/site-packages/kubernetes/client/models/v1_endpoint_slice.py", line 147, in endpoints
    raise ValueError("Invalid value for `endpoints`, must not be `None`")  # noqa: E501
ValueError: Invalid value for `endpoints`, must not be `None`

What you expected to happen:

Expected a list of endpointslices to be returned.

How to reproduce it (as minimally and precisely as possible):

Create an endpointslice with endpoints as null. See bottom of issue for example of null endpoints.

Execute this code on any kubernetes cluster.

from kubernetes import client, config

config.load_kube_config()  # For local development

discovery_api = client.DiscoveryV1Api()
endpoint_slices = discovery_api.list_endpoint_slice_for_all_namespaces()

for endpoint_slice in endpoint_slices.items:
    print(endpoint_slice.metadata.name)

Anything else we need to know?:

Name: kubernetes
Version: 28.1.0
Summary: Kubernetes python client
Home-page: https://github.com/kubernetes-client/python
Author: Kubernetes
Author-email:
License: Apache License Version 2.0
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: certifi, google-auth, oauthlib, python-dateutil, pyyaml, requests, requests-oauthlib, six, urllib3, websocket-client

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.14
WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1

Noteworthy that if I debug this a bit, I do see that I am receiving the EndpointSliceList correctly from the apiserver, but the client is struggling to do...something with it.

====
After further investigation, it appears related to endpointslices with null endpoints for example. If I debug down, I find that it's failing to deserialize on this particular endpointslice in my cluster.

k get endpointslice node-local-dns-rfnd6 -o yaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints: null
kind: EndpointSlice
metadata:
  creationTimestamp: "2023-02-24T10:19:02Z"
  generateName: node-local-dns-
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    helm.toolkit.fluxcd.io/name: node-local-dns
    helm.toolkit.fluxcd.io/namespace: kube-system
    k8s-app: node-local-dns
    kubernetes.io/service-name: node-local-dns
  name: node-local-dns-rfnd6
  namespace: kube-system
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: node-local-dns
    uid: 862d5943-0a42-42da-80f3-74f2cfe8d23d
  resourceVersion: "1979322046"
  uid: a73a2d67-4db4-4a9d-a0f5-1ae6ca78631a
ports: null

[showjason]

/assign

[showjason]

@bpinske from the Kubernetes source code, we can see the endpoints must not be empty, so your endpointslice node-local-dns-rfnd6 is incorrect.

[bpinske]

maybe this is my ignorance showing, but how does that say endpoints cannot be null?

I have many endpointslices it seems which have null endpoints. Including very old stuff like Helm Tiller which I haven't used in years. So it definitely happens regularly and as a normal thing. I don't think the python client library should fail to handle this when the Go library does handle it properly.

Further, I ended up just writing my tool in golang to unblock mysel. I do have the desired and expected behaviour when using Go.

Here is a simple Go version which does have the expected behaviour and does handle null endpoints.

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"

	_ "k8s.io/client-go/plugin/pkg/client/auth/oidc"
)

const (
	kubeconfigEnvVar = "KUBERNETES_SERVICE_HOST"
)

func main() {
	config, err := getKubeConfig()
	if err != nil {
		fmt.Printf("Failed to get Kubernetes config: %s\n", err)
		os.Exit(1)
	}

	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		fmt.Printf("Failed to create Kubernetes client: %s\n", err)
		os.Exit(1)
	}

	printEndpointSlices(clientset)
}

func getKubeConfig() (*rest.Config, error) {
	if _, exists := os.LookupEnv(kubeconfigEnvVar); exists {
		fmt.Println("Running in cluster detected")
		return rest.InClusterConfig()
	}
	fmt.Println("Running outside of cluster detected")

	kubeconfigPath := filepath.Join(homedir.HomeDir(), ".kube", "config")
	return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}

func printEndpointSlices(clientset *kubernetes.Clientset) {
	fmt.Println("Fetching EndpointSlices...")
	slices, err := clientset.DiscoveryV1().EndpointSlices("").List(context.Background(), metav1.ListOptions{})
	if err != nil {
		fmt.Printf("Error fetching EndpointSlices: %s\n", err)
		os.Exit(1)
	}

	for _, slice := range slices.Items {
		fmt.Printf("Namespace: %s, Name: %s, Endpoints:\n", slice.Namespace, slice.Name)
		for _, endpoint := range slice.Endpoints {
			fmt.Printf("  - Addresses: %v\n", endpoint.Addresses)
		}
	}
}

outputs

Namespace: default, Name:populatedendpointslice-8qjxn, Endpoints:
  - Addresses: [10.2.110.161]
  - Addresses: [10.2.206.229]
Namespace: default, Name: emptyendpointslice-7pwhd, Endpoints:

[showjason]

Hi @bpinske , sorry, it's my stupid mistake, the source code means field endpoints is required, but not means its value must not be empty.
I also tried to create an endpointslice with empty endpoints via kubectl and succeeded.
Let me open a PR to fix this bug

[showjason]

Hi @roycaihw , this endpoints is not validated properly, it's generated by OpenAPI Generator automatically and not allowed to edit manually, hence I check the Kubernetes source code and find that there are specific validation functions() validating the endpointslices and endpoints in validation.go,but they are not generated by OpenAPI Generator as expected.
If my understanding is correct, do you think this bug will be fixed in the coming release?

[showjason]

Issue #1662 can resolve this problem by disabling the client side validation