Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to interact with cluster through Teleport #3242

Open
nicolas-goudry opened this issue Mar 22, 2024 · 10 comments
Open

Unable to interact with cluster through Teleport #3242

nicolas-goudry opened this issue Mar 22, 2024 · 10 comments

Comments

@nicolas-goudry
Copy link

nicolas-goudry commented Mar 22, 2024 •
edited

Describe the bug

Using a cluster with authn via Teleport, it is not possible to interact with the cluster.

This is working fine with kubectl and after testing with the Python client it works too. This seems to only happen with the Java client (I didn’t test other clients though).

Client Version

20

Kubernetes Version

1.28.6

Java Version

17

To Reproduce

  • configure your cluster with Teleport
  • generate a kubeconfig through tsh

Edit the KubeConfigFileClientExample:

diff --git a/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java b/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
index 7a0ba4dcf..99a7cc75e 100644
--- a/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
+++ b/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
@@ -16,12 +16,13 @@ import io.kubernetes.client.openapi.ApiClient;
 import io.kubernetes.client.openapi.ApiException;
 import io.kubernetes.client.openapi.Configuration;
 import io.kubernetes.client.openapi.apis.CoreV1Api;
-import io.kubernetes.client.openapi.models.V1Pod;
-import io.kubernetes.client.openapi.models.V1PodList;
+import io.kubernetes.client.openapi.models.V1Namespace;
+import io.kubernetes.client.openapi.models.V1ObjectMeta;
 import io.kubernetes.client.util.ClientBuilder;
 import io.kubernetes.client.util.KubeConfig;
 import java.io.FileReader;
 import java.io.IOException;
+import java.nio.file.Paths;
 
 /**
  * A simple example of how to use the Java API from an application outside a kubernetes cluster
@@ -37,10 +38,12 @@ public class KubeConfigFileClientExample {
     // file path to your KubeConfig
 
     String kubeConfigPath = System.getenv("HOME") + "/.kube/config";
+    KubeConfig kubeconfig = KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath));
+    kubeconfig.setFile(Paths.get(kubeConfigPath).toFile());
 
     // loading the out-of-cluster config, a kubeconfig from file-system
     ApiClient client =
-        ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath))).build();
+        ClientBuilder.kubeconfig(kubeconfig).build();
 
     // set the global default api-client to the in-cluster one from above
     Configuration.setDefaultApiClient(client);
@@ -48,11 +51,11 @@ public class KubeConfigFileClientExample {
     // the CoreV1Api loads default api-client from global configuration.
     CoreV1Api api = new CoreV1Api();
 
-    // invokes the CoreV1Api client
-    V1PodList list =
-        api.listPodForAllNamespaces().execute();
-    for (V1Pod item : list.getItems()) {
-      System.out.println(item.getMetadata().getName());
-    }
+    V1Namespace ns = new V1Namespace();
+    V1ObjectMeta meta = new V1ObjectMeta();
+    meta.setName("test");
+    ns.setMetadata(meta);
+    V1Namespace result = api.createNamespace(ns).pretty("true").execute();
+    System.out.println(result.getMetadata().getName());
   }
 }

Run the example:

cd examples/examples-release-20
mvn -X clean install exec:java -Dexec.mainClass="io.kubernetes.client.examples.KubeConfigFileClientExample"

Expected behavior

The namespace is created.

KubeConfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: REDACTED
    tls-server-name: REDACTED
  name: REDACTED
contexts:
- context:
    cluster: REDACTED
    extensions:
    - extension: sandbox
      name: teleport.kube.name
    user: REDACTED
  name: REDACTED
current-context: REDACTED
kind: Config
preferences: {}
users:
- name: REDACTED
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kube
      - credentials
      - --kube-cluster=sandbox
      - --teleport-cluster=REDACTED
      - --proxy=REDACTED
      command: tsh
      env: null
      provideClusterInfo: false

Server (please complete the following information):

  • OS: Linux
  • Environment: system
  • Cloud: Azure/Teleport

Additional context

My user has limited permissions, but I should be able to create a namespace:

$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
namespaces.*                                    []                  []               [get list watch create delete]
nodes                                           []                  []               [get list watch]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]
                                                [/version]          []               [get]
@brendandburns
Copy link
Contributor

What exception are you seeing?

@nicolas-goudry
Copy link
Author

nicolas-goudry commented Mar 23, 2024 •
edited

Here is the exception I’m seeing:

Caused by: io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Fri, 22 Mar 2024 16:09:32 GMT], x-content-type-options=[nosniff]}

It seems that somehow the authentication process which uses Teleport is not working and therefore the request is not authorized by the API.

@brendandburns
Copy link
Contributor

Without access to a teleport cluster it's kind of hard to debug this.

The relevant exec code is here:

java/util/src/main/java/io/kubernetes/client/util/KubeConfig.java

Line 269 in ef07a5b

private Map<String, String> getCredentialsViaExecCredential(Map<String, Object> execMap) {

Can you turn on debug logging and see what it seems to be doing?

@nicolas-goudry
Copy link
Author

nicolas-goudry commented Mar 24, 2024 •
edited

Can you turn on debug logging and see what it seems to be doing?

By “debug logging” I guess you meant running maven with the -X flag. I also used client.setDebugging(true).

Here are the logs:

Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: --> POST https://REDACTED.teleport.sh/api/v1/namespaces
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 57
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Accept: application/json
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: User-Agent: Kubernetes Java Client/20.0.0-SNAPSHOT
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: 
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: {"metadata":{"annotations":{},"labels":{},"name":"test"}}
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: --> END POST (57-byte body)
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: <-- 403 Forbidden https://REDACTED.teleport.sh/api/v1/namespaces (486ms)
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: text/plain; charset=utf-8
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: X-Content-Type-Options: nosniff
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Date: Sun, 24 Mar 2024 07:54:17 GMT
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 10
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: 
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Forbidden

Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (10-byte body)
[WARNING] 
io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at io.kubernetes.client.openapi.ApiClient.handleResponse (ApiClient.java:1116)
    at io.kubernetes.client.openapi.ApiClient.execute (ApiClient.java:1029)
    at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespaceWithHttpInfo (CoreV1Api.java:9213)
    at io.kubernetes.client.openapi.apis.CoreV1Api.access$19300 (CoreV1Api.java:77)
    at io.kubernetes.client.openapi.apis.CoreV1Api$APIcreateNamespaceRequest.execute (CoreV1Api.java:9307)
    at io.kubernetes.client.examples.KubeConfigFileClientExample.main (KubeConfigFileClientExample.java:59)
    at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0 (ExecJavaMojo.java:283)
    at java.lang.Thread.run (Thread.java:833)
[DEBUG] interrupting thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample]
[DEBUG] joining on thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample]
[WARNING] thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=io.kubernetes.client.examples.KubeConfigFileClientExample,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:803)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:331)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.620 s
[INFO] Finished at: 2024-03-24T08:54:32+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.1:java (default-cli) on project client-java-examples-release-20: An exception occurred while executing the Java class. Message: Forbidden
[ERROR] HTTP response code: 403
[ERROR] HTTP response body: Forbidden
[ERROR] 
[ERROR] HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.1:java (default-cli) on project client-java-examples-release-20: An exception occurred while executing the Java class. Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occurred while executing the Java class. Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:354)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at io.kubernetes.client.openapi.ApiClient.handleResponse (ApiClient.java:1116)
    at io.kubernetes.client.openapi.ApiClient.execute (ApiClient.java:1029)
    at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespaceWithHttpInfo (CoreV1Api.java:9213)
    at io.kubernetes.client.openapi.apis.CoreV1Api.access$19300 (CoreV1Api.java:77)
    at io.kubernetes.client.openapi.apis.CoreV1Api$APIcreateNamespaceRequest.execute (CoreV1Api.java:9307)
    at io.kubernetes.client.examples.KubeConfigFileClientExample.main (KubeConfigFileClientExample.java:59)
    at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0 (ExecJavaMojo.java:283)
    at java.lang.Thread.run (Thread.java:833)

Let me know if I should enable any other option.

@brendandburns
Copy link
Contributor

brendandburns commented Mar 25, 2024 •
edited

Can you try it adding -Dorg.slf4j.simpleLogger.defaultLogLevel=debug There are some debug level logs in the kubeconfig loader that will (hopefully) provide more info.

I'm more interested in the logs when the kubeconfig is loaded, vs. the logs when the call is made.

@nicolas-goudry
Copy link
Author

I added the option, ending up with this command:

mvn -X clean install exec:java -Dexec.mainClass="io.kubernetes.client.examples.KubeConfigFileClientExample" -Dorg.slf4j.simpleLogger.defaultLogLevel=debug

There aren’t any further logs than the ones I sent earlier. I tried with -Dorg.slf4j.simpleLogger.defaultLogLevel=trace too, without success.

@brendandburns
Copy link
Contributor

I'm specifically trying to see if these log lines are loading:

java/util/src/main/java/io/kubernetes/client/util/KubeConfig.java

Line 307 in ef07a5b

log.debug("Obtained a client certificate from {}", command);

java/util/src/main/java/io/kubernetes/client/util/KubeConfig.java

Line 317 in ef07a5b

log.debug("Obtained a token from {}", command);

My bet is that the -Dorg.sl4j... is going to Maven, not to the java exec, but I'm not certain.

@nicolas-goudry
Copy link
Author

nicolas-goudry commented Mar 26, 2024 •
edited

My bet is that the -Dorg.sl4j... is going to Maven, not to the java exec, but I'm not certain.

That’s indeed what is happening here… I tried several solutions to pass the property down to the java exec, without any success.

In the end, I decided to manually edit the code to use the info level and use examples-release-latest to do my test. I know it’s not very conventional, but at least it worked 😅

Here are the relevant logs you were seeking:

09:25:07.541 [io.kubernetes.client.examples.KubeConfigFileClientExample.main()] INFO io.kubernetes.client.util.KubeConfig -- Resolved /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh to /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh
09:25:07.642 [io.kubernetes.client.examples.KubeConfigFileClientExample.main()] INFO io.kubernetes.client.util.KubeConfig -- Obtained a client certificate from /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh

So in the end it seems that the authentication part is working fine with the exec command… The issue seems to be elsewhere.

@brendandburns
Copy link
Contributor

Given that it is loading the cert, this all gets injected into the request here:

https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/util/credentials/KubeconfigAuthentication.java#L45

How long do these teleport certificates last? The authenticator only loads it at config loading time, and doesn't reload ever, so if there is a short expiration time that might cause it.

@nicolas-goudry
Copy link
Author

nicolas-goudry commented Apr 3, 2024 •
edited

I checked by manually running the tsh command which generates the certificate: it is valid for 12 hours.

I wondered if this issue could somehow be linked to the lack of support for tls-server-name, but I get the same error without it (same kubeconfig, remove certificate-authority-data and tls-server-name and add insecure-skip-tls-verify: true).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brendandburns @nicolas-goudry