WebSocketHandshakeException: null occured when exec into a pod

[SrihithaG29]

Hi @manusa / Team,

We are trying to use exec functionality of Fabric8 kubernetes client.
Created a application to implement the same to exec into a pod but it fails with WebSocketHandshakeException: null,
Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'\n\tat

Using 6.5.0 version kubernetes-client, kubernetes-client-api, kubernetes-httpclient-okhttp.

Please find the implementation and the stacktrace below :

import java.io.ByteArrayOutputStream;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;

import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientBuilder;
import io.fabric8.kubernetes.client.KubernetesClientException;
import io.fabric8.kubernetes.client.dsl.ExecListener;
import io.fabric8.kubernetes.client.dsl.ExecWatch;


@SpringBootApplication
public class ApplicationTest {
    private static final Logger logger = LoggerFactory.getLogger(ApplicationTest.class);
    private static final CountDownLatch execLatch = new CountDownLatch(1);

    public static void main(final String[] args) {
        SpringApplication.run(ApplicationTest.class, args);
        String podName = "somepod";
        String namespace = "namespace";

        try (final KubernetesClient k8s = new KubernetesClientBuilder().build()) {
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            ByteArrayOutputStream error = new ByteArrayOutputStream();
            logger.info("pods get :: {}",k8s.pods().inNamespace(namespace).withName(podName).get());
            Pod pod = k8s.pods().inNamespace(namespace).withName(podName).get();
            if(pod != null) {
                logger.info("If-start");
                ExecWatch watch = k8s.pods().inNamespace(namespace).withName(podName).redirectingInput().redirectingOutput()
                        .redirectingError().exec("/bin/bash","-c","ls");
                watch.close();
            }
        }       
        catch (KubernetesClientException e) {
              logger.error("ExceptionOccured : {}",e);
              logger.error("stacktrace");
              e.printStackTrace();
        }
    }   
}

When I deploy the application in a kubernetes cluster with version

kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:14:09Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

{"timestamp":"2023-11-06T08:27:51.576Z","version":"0.3.0","message":"Exec Failure","logger":"io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener","thread":"OkHttp https://10.96.0.1/...","stack_trace":"io.fabric8.kubernetes.client.http.WebSocketHandshakeException: null\n\tat io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:72)\n\tat okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)\n\tat okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)\n\tat okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: java.lang.Throwable: waiting here\n\t\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:173)\n\t\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:184)\n\t\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.setupConnectionToPod(PodOperationsImpl.java:321)\n\t\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:276)\n\t\tat com.ApplicationTest.main(ApplicationTest.java:72)\n\t\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\t\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\t\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\t\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\t\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\t\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:95)\n\t\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\t\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\nCaused by: java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'\n\tat okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess$okhttp(RealWebSocket.kt:224)\n\tat okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:170)\n\t... 4 common frames omitted\n","service_id":"eric-oss-osm","severity":"error"}
{"timestamp":"2023-11-06T08:27:51.576Z","version":"0.3.0","message":"ExceptionOccured : {}","logger":"com.ApplicationTest","thread":"main","stack_trace":"io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.\n\tat io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)\n\tat io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)\n\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:174)\n\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:184)\n\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.setupConnectionToPod(PodOperationsImpl.java:321)\n\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:276)\n\tat com.ApplicationTest.main(ApplicationTest.java:72)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:95)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\nCaused by: io.fabric8.kubernetes.client.http.WebSocketHandshakeException: null\n\tat io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:72)\n\tat okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)\n\tat okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)\n\tat okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: java.lang.Throwable: waiting here\n\t\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:173)\n\t\tat io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:184)\n\t\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.setupConnectionToPod(PodOperationsImpl.java:321)\n\t\tat io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:276)\n\t\tat com.ApplicationTest.main(ApplicationTest.java:72)\n\t\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\t\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\t\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\t\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\t\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\t\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:95)\n\t\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\t\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\nCaused by: java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'\n\tat okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess$okhttp(RealWebSocket.kt:224)\n\tat okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:170)\n\t... 4 common frames omitted\n","service_id":"eric-oss-osm","severity":"error"}
{"timestamp":"2023-11-06T08:27:51.581Z","version":"0.3.0","message":"stacktrace","logger":"com.ApplicationTest","thread":"main","service_id":"eric-oss-osm","severity":"error"}
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
        at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:174)
        at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:184)
        at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.setupConnectionToPod(PodOperationsImpl.java:321)
        at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:276)
        at com.ApplicationTest.main(ApplicationTest.java:72)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: io.fabric8.kubernetes.client.http.WebSocketHandshakeException
        at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:72)
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.kt:592)
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:174)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:173)
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:184)
                at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.setupConnectionToPod(PodOperationsImpl.java:321)
                at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:276)
                at com.ApplicationTest.main(ApplicationTest.java:72)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:568)
                at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
                at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
                at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
                at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess$okhttp(RealWebSocket.kt:224)
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:170)
        ... 4 more

We are not sure what exactly the issue here is as it says null, we have created a role, rolebinding and referenced it to the service account but still we could see the issue. Please help us here to understand more about the issue.

Thanks in advance.

[manusa]

Your Pod doesn't seem to have access to the Kubernetes API.
Usually, Kubernetes Client automatically takes care of inferring your auth credentials and settings by checking the mounted service account files. It's very likely that there's an issue with this (Especially since you mention you've created a role, role binding, etc.).

Make sure you're following:

• https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod
• https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

You should try to TTY into the Pod and make sure everything is in place.

[SrihithaG29]

Hi @manusa

How to ensure that the pod has the access to the Kubernetes API ?
Earlier when I implemented the same code, I've seen that it is not able to list the resources with the mounted ServiceAccount.
So after creating the Role & Rolebinding with the resources as pods and resources, we were able to list the pods
i.e, k8s.pods().inNamespace(namespace).withName(podName).get();
I've even tried with TTY but same issue persists.
Please let me know what can I check further.

Thanks In advance.

[manusa]

Check the links I shared in my previous response. It's clear that you have problems with your Roles, Role Bindings, and Role Binding Access Control (RBAC).
Probably the role you configured doesn't have complete access to the required resources.

RBAC allows you to provide fine-grained control about the HTTP endpoints/resources and operations a user is allowed to perform.