Jetty 11 WebSocket Client: SSL/TLS authentication using client certificate

[manusa]

Jetty version: 11.0.9

Java version: 11.0.10

Question

I'm working on an HTTPClient implementation for Fabric8 Kubernetes Client based on Eclipse Jetty.

You can check the work in progress in the following PR: fabric8io/kubernetes-client#4180

This implementation requires both, the jetty-client and the jetty-websocket-client.

To set up SSL support for both clients I'm using the following snippet:

// JettyHttpClientBuilder#build
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
sslContext.init(ourKeyManagers, ourTrustManagers, new SecureRandom());
final var sslContextFactory = new SslContextFactory.Client();
sslContextFactory.setSslContext(sslContext);
final var clientConnector = new ClientConnector();
clientConnector.setSslContextFactory(sslContextFactory);
final var httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));
final var webSocketClient = new WebSocketClient(httpClient);

This code works fine in both clients(HTTP and websocket) when just dealing with the TLS transport layer.

The HTTP client works fine when a client certificate is used for authentication too. However, when using the websocket client using the client certificate, we get an Upgrade Exception with a Forbidden status code.

I'm not sure if there might be an issue with our configuration, or this might be a bug.

[sbordet]

Your code is correct.

If you can reproduce the case, can you enable DEBUG logs on the server, so we can understand why the request failed?

[manusa]

These are the debug logs https://gist.github.com/manusa/9d4ed942d96909c17c26169589bcaf60, line 395 has the forbidden response.

For some additional context, the endpoint I'm trying to connect is https://192.168.39.24:8443/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsample-watch-pod&allowWatchBookmarks=true&watch=true, however, we're changing the protocol to wss before passing it to the websocket client; the resulting URI would be wss://192.168.39.24:8443/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsample-watch-pod&allowWatchBookmarks=true&watch=true. I'm not sure if this might affect the way the client certificate is selected.

[sbordet]

You linked the client logs. Need the server logs to understand why the server had a TLS failure.

[manusa]

Oh, sorry, I missed the server part in your first comment.

I don't have access to the server logs. This is a Minikube Kubernetes server , and I'm not sure this kind of logging detail (or any logging at all) can be enabled (currently checking this).

The thing is that for the HttpClient, this authentication works and the server accepts the client's requests. However, the same doesn't for the WebSocketClient. The SSL connection is established, but then the request fails with the mentioned 403 Forbidden status. As I said, my suspicion is that for some reason the client certificate is not used for authentication in this case.

[sbordet]

@manusa it could well be that the server is configured differently, because the client is configured the same.

Jetty does not do much with respect to TLS, it just delegates to the JDK, so I doubt that "for some reason the client certificate is not used for authentication in this case" due to client problems -- I would concentrate on the server configuration first.

[manusa]

The thing is that the same procedures work both with the OkHttp client, and the JDK (11+) HttpClient. Maybe we're missing an extra-header that those clients add automatically, or something else 🤷. I'm 90% certain that this has to do with the authentication procedure, since SSL works when no client certificate is required for authentication.

The problem is that debugging this is really hard due to the TLS layer (tried wiresharking, but was unable to setup the decryption keys).
I'm checking now if I can enable auditing and better logging for the kube-api-server, to see if I can get a clarification on the 403 status.

[sbordet]

You can use -Djavax.net.debug=all on the server to know more.

Also, it may not be a TLS issue at all, maybe just a WebSocket issue. TLS is fine, but for some reason the server does not like the WebSocket upgrade.

[manusa]

You can use -Djavax.net.debug=all on the server to know more.

Maybe this is not clear (or I'm not following). The server is a Kubernetes server (more specifically a Minikube instance), this flag is not applicable.

[sbordet]

I don't know how to help you more if you don't get something from the server.

How do you know that OkHttp and JDK's HttpClient work with client certificates?
Maybe they don't send the client certificates, so only plain TLS works, which works in Jetty too.

How do you configure the server to require client certificates?

So far nothing points to Jetty.
The way you setup TLS in Jetty's client is based on JDK's SSLContext. Jetty uses this JDK class and does not do much else.

How do you setup the client KeyStore with the certificates? Is the client certificate properly signed? Is the server setup correctly do verify the client certificates?

Have you tried to not force TLS 1.2?

[manusa]

How do you know that OkHttp and JDK's HttpClient work with client certificates?
Maybe they don't send the client certificates, so only plain TLS works, which works in Jetty too.

Our project is based on OkHttp and we're now providing additional client implementations. We thought that the native Jdk HttpClient and Jetty would be good choices.
Both these clients (OkHttp and JDK) work on our E2E test suite that runs on Minikube (which requires the client certificate for authentication). My current WIP implementation works for all cases except this one (and only fails on the WebSocket client requests).
So I'm certain that there is some issue or difference with how Jetty handles the WebSocket upgrade.

I'm thinking that that the http-to-ws URI protocol flipping we do to avoid the Jetty websocket connect check might have something to do (this is reversed in the JDK implementation when performing the handshake). (Update: I prevented the check in a debug session and the result is still NOT OK with an https protocol)

How do you configure the server to require client certificates?

I don't, this is part of Minikube, the initial user Minikube creates can authenticate with some certificates.

How do you setup the client KeyStore with the certificates? Is the client certificate properly signed? Is the server setup correctly do verify the client certificates?

This is a complex procedure, but as I said, same thing works for OkHttp and JDK. The relevant part to Jetty is how we setup the keyManagers and trustManagers in an SSLContext and pass it to the SslContextFactory (shown in the description snippet). You already confirmed that this was fine.

Have you tried to not force TLS 1.2?

As I said, without TLS it does work, and with TLS and no authentication, it works too. The problem is just when a client certificate is required for authentication.

[manusa]

I think I managed to solve the issue. Thanks for your help and involvement @sbordet 🙌

The WebSocket upgrade request is missing the Origin header, apparently the rest of clients add this header automatically.

I'll add more details and close the issue once I'm sure everything works.

[manusa]

As usual in this cases, the problem exists between keyboard and chair.

We're adding the Origin header ourselves, but I was not propagating them to the ClientUpgradeRequest 🤦

Thanks again for your help!