Fabric8 Kubernetes client is not working with Teleport

[fslevoaca-ionos]

Describe the bug

Hi,

It seems Fabric8 Kubernetes client doesn't work with teleport.

Fabric8 Kubernetes Client version

6.11.0

Steps to reproduce

I have the following teleport Kube config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://teleport.test.com:443
    tls-server-name: kube-teleport-proxy.teleport.test.com
  name: tp.pf.test.net
contexts:
- context:
    cluster: tp.pf.test.net
    user: tp.pf.test.net.net-ag-s01
  name: tp.pf.test.net.net-ag-s01
current-context: tp.pf.test.net.net-ag-s01
kind: Config
preferences: {}
users:
- name: tp.pf.test.net.net-ag-s01
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kube
      - credentials
      - --kube-cluster=ag-s01
      - --teleport-cluster=tp.pf.test.net.net-ag-s01
      - --proxy=teleport.test.com:443
      command: /usr/local/bin/tsh
      env: null
      provideClusterInfo: false

kubectl command with the Kubeconfig from above works fine.
However, using the fabric8 Kubernetes client with the same Kubeconfig from above, it doesn't work.
I get 403 Forbidden message:

Caused by: java.lang.ExceptionInInitializerError: Exception io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://teleport.test.com/api/v1/namespaces/testns/secrets/tenant-credentials. Message: Forbidden. [in thread "main"]
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:98)

Expected behavior

Java Kubernetes client should also work with teleport configuration.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.25.3@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

[main] DEBUG io.fabric8.kubernetes.client.utils.HttpClientUtils - Using httpclient io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory factory
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - -HTTP START-
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - > GET https://teleport.test.com:443/api/v1/namespaces/testns/secrets/tenant-credentials
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - > User-Agent: fabric8-kubernetes-client/6.11.0
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - < 403 Forbidden
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - < content-length: 10
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - < content-type: text/plain; charset=utf-8
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - < date: Wed, 03 Apr 2024 09:02:03 GMT
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - < x-content-type-options: nosniff
[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - Forbidden

[OkHttp Dispatcher] TRACE io.fabric8.kubernetes.client.http.HttpLoggingInterceptor - -HTTP END-
[OkHttp Dispatcher] DEBUG io.fabric8.kubernetes.client.dsl.internal.OperationSupport - Exception convertion response to Status
java.lang.IllegalArgumentException: Cannot construct instance of `io.fabric8.kubernetes.api.model.Status` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('Forbidden')
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4624)
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4565)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.parseYaml(KubernetesSerialization.java:275)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:251)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:341)
	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:326)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.createStatus(OperationSupport.java:612)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `io.fabric8.kubernetes.api.model.Status` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('Forbidden')
 at [Source: UNKNOWN; byte offset: #UNKNOWN]

Additional context

Might be related to #5292

[shawkins]

Did you try the workaround from #5292 (comment)

[fslev]

The workaround seems to work. But in my case it seems it is not enough since I am also making use of Fabric8 Kubernetes client port forwarding feature which doesn't work with kubectl proxying.

[shawkins]

@fslev that's unfortunate. Seems like at least our jetty client will need to be updated to support this.