DOCUMENT how to have NATS behind TLS terminating proxy

[boris-senapt]

Observed behavior

I am attempting to deploy NATS in AWS EKS with an NLB running a TLS terminating reverse proxy.

Clients inside the cluster will talk to NATS using plaintext.

I following the guidance in the documentation to set up the Java client but I am seeing

Caused by: java.io.IOException: SSL connection wanted by client.
	at io.nats.client.impl.NatsConnection.upgradeToSecureIfNeeded(NatsConnection.java:606) ~[jnats-2.17.6.jar:2.17.6]
	at io.nats.client.impl.NatsConnection.lambda$tryToConnect$2(NatsConnection.java:474) ~[jnats-2.17.6.jar:2.17.6]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

Expected behavior

I expect the Java NATS client to connect to the NATS server behind an TLS terminating reverse proxy with the documented workaround.

Server and client version

Server is installed from Helm chart 1.1.11 which deploys nats:2.10.14-alpine sha 81f36bfe9dfef7cd3768abaf55fc309123cf6bf1cb0d8305a6700ff36034b93bz

Client is io.nats:jnats:2.17.6

Host environment

Not sure it's relevant, but test client is Mac OS Sonoma 14.4.1 Apple M2 with GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30)

Steps to reproduce

Deploy NATS to AWS EKS using the specified helm chart, with the AWS Ingress controller and the following service annotations

    service:
      merge:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
            service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "<>"
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '4222'
            service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: '8222'
            service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/healthz"

This should provision and NLB with TLS termination on port 4222 available on an external IP address with a DNS name.

---

Set up the NATS config as described in the documentation and attempt to connect.

---

As the server does not have TLS enabled nor required - it cannot do as it does not run TLS and clients inside the cluster communicate in plain text - and the client requires TLS due to:

// io.nats.client.Options#isTLSRequired
    public boolean isTLSRequired() {
        return tlsFirst || this.sslContext != null;
    }

Once the NATS client connects to the server, the following check fails

// io.nats.client.impl.NatsConnection#upgradeToSecureIfNeeded
if (isTLSRequired && !serverInfo.isTLSRequired()) {
    throw new IOException("SSL connection wanted by client.");
}

[boris-senapt]

As a test I made the following change to the library and the connection works

diff --git a/src/main/java/io/nats/client/Options.java b/src/main/java/io/nats/client/Options.java
index a3f94613..d8f6a801 100644
--- a/src/main/java/io/nats/client/Options.java
+++ b/src/main/java/io/nats/client/Options.java
@@ -2108,7 +2108,7 @@ public class Options {
      * @return true if there is an sslContext for these Options, otherwise false, see {@link Builder#secure() secure()} in the builder doc
      */
     public boolean isTLSRequired() {
-        return sslContext != null;
+        return false;
     }

     /**

So what is needed is an option to force the client not to require TLS even if sslContext is set - maybe an option called something like enableReverseProxy()?

---

As an aside, the option tlsFirst() allows the connection to work without the TlsSocketDataPort hack.

[boris-senapt]

Okay - it seems using .secure().tlsFirst() works for a TLS reverse proxy!

[scottf]

Yes, TLS first doesn't do the same checking.

[boris-senapt]

Yes, TLS first doesn't do the same checking.

Would it be worth keeping this ticket or creating a new one to update the docs? Because they had me chasing my tail for a while there; and I only stumbled on the right answer by reading the code.

[scottf]

We have only recently had requests for dealing with reverse proxy, in fact before we explicitly did not want to support it. I will re-open and change this issue to a documentation issue. As of 2.17.7-SNAPSHOT, we can now support tls proxy termination where the client to proxy is tls, but the proxy to server is insecure.