NettyNioAsyncHttpClient not respecting replacements of JSSE Providers (e.g. Bouncy Castle FIPS) by using own CipherSuites

[jakobmoellerdev]

Hey everyone! I am currently setting up the AWS SDK to run with SQS and S3. However my service is running in a FIPS-compliant, standardised Docker environment. Unfortunately, this also means exchanging used Security Providers (OpenSSL is excluded from the classpath through Gradle) under the Hood of JVM and thus also the SDK.

The problem occurs when trying to spin up an SdkAsyncHttpClient with a corresponding NettyNioAsyncHttpClient as I am running in Spring Boot with Netty already for other operations and I want to keep things consistend and non-blocking.

Describe the bug

When trying to establish the connection via Netty in the SDK to AWS Services, Ciphers cannot be negotiated correctly and the SSL Handshake fails.

{"@timestamp": "2020-11-23T16:28:40.471+0000", "level": "INFO", "class": "org.bouncycastle.jsse.provider.ProvTlsServer", "description": "Server raised fatal(2) handshake_failure(40) alert: Failed to process record org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40) 
at org.bouncycastle.tls.AbstractTlsServer.getSelectedCipherSuite(Unknown Source) 
at org.bouncycastle.jsse.provider.ProvTlsServer.getSelectedCipherSuite(Unknown Source) 
at org.bouncycastle.tls.TlsServerProtocol.generateServerHelloMessage(Unknown Source) 
at org.bouncycastle.tls.TlsServerProtocol.handleHandshakeMessage(Unknown Source)}

Expected Behavior

I was expecting a successful handshake with the AWS Server performed with using Cipher Suites common to both the Bouncy Castle FIPS Library responsible for TLS (JSSE) through Netty which under the Hood usually defaults to the correct SSL Settings if configured through the JVM.

Whenever trying to establish a connection, a cipher that is compatible should be negotiated and the TLS Handshake completed. I can replicate a successful handshake with a Netty Client spun up by myself in my service and negotiating with a different non-aws service (e.g. A MySQL through Hikari):

Client notified of selected protocol version: TLSv1.2
Client notified of selected cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

I am using the same default SSL Context and Provider that I am expecting to be drawn in to the SDK from the JVM as well.

Current Behavior

in the case of the above failure, I was accessing us-west-2.queue.amazonaws.com

According to the SSL Analysis here:
https://www.ssllabs.com/ssltest/analyze.html?d=us-west-2.queue.amazonaws.com

and my SSL Configuration below

Protocols
SSLContext.getDefault().getDefaultSSLParameters().getProtocols()
[TLSv1.2, TLSv1.1, TLSv1]
CipherSuites
SSLContext.getDefault().getDefaultSSLParameters().getCipherSuites()
[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]

From my limited knowledge (I am not exactly a security expert), there should be several ciphers listed that are supported for both my client and server for negotiation, indicating an issue in the SDK

What I am seeing however, is the following:
[id: OMITTED] REGISTERED
[id: OMITTED] CONNECT
Client raised warning(1) user_canceled(90) alert: User canceled handshake
Client raised warning(1) close_notify(0) alert: Connection closed
[id: OMITTED, L:/OMITTED- R:sqs.us-west-2.amazonaws.com/OMITTED:443] USER_EVENT: SslHandshakeCompletionEvent(java.lang.IllegalStateException: No usable cipher suites enabled)

Steps to Reproduce

The JVM is setup like this in a Dockerfile

ENV JAVA_OPTS --show-version \ -XshowSettings:vm \ -XX:+DisableExplicitGC \ -Dfile.encoding=UTF-8 \ -Dorg.bouncycastle.fips.approved_only=true \ -Xbootclasspath/a:/PATHTOJARS/bc-fips-${BC_VERSION}.jar \ -Xbootclasspath/a:/PATHTOJARS/bctls-fips-${BC_TLS_VERSION}.jar"

I am using these Options to generate my Keystore

ENV KEYTOOL_OPTS "-storetype BCFKS \ -providerpath /PATHTOJARS/bc-fips-${BC_VERSION}.jar \ -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider"

and some changes to the java.security file for setting up the JVM correctly.

RUN sed -i 's/\(security\.provider\.[0-9]\+\)=\(.*\)/##\1=\2/g' java.security
RUN sed -i 's/\(ssl\.KeyManagerFactory\.algorithm\)=.*/\1=PKIX/g' java.security
RUN sed -i 's/\(securerandom\.source\)=.*/\1=file:\/dev\/urandom/g' java.security
RUN sed -i 's/\(securerandom\.strongAlgorithms\)=.*/\1=NativePRNGNonBlocking:SUN,DRBG:SUN/g' java.security
RUN printf '\nsecurity.provider.1=BCFIPS\nsecurity.provider.2=BCJSSE fips:BCFIPS\nsecurity.provider.3=SUN\n' >> java.security

as well as

-Dio.netty.handler.ssl.noOpenSsl=true \ -Djavax.net.ssl.trustStoreProvider=BCFIPS \ -Djavax.net.ssl.trustStoreType=BCFKS \ -Djavax.net.ssl.trustStorePassword=changeit \ -Djavax.net.debug=ssl,handshake \

If you want to configure the providers by Code you can use this:

compile group: 'org.bouncycastle', name: 'bc-fips', version: '1.0.2'
compile group: 'org.bouncycastle', name: 'bctls-fips', version: '1.0.10'

private static void makeOurServiceFIPSCompliant() {
        Security.insertProviderAt(new BouncyCastleFipsProvider(),1); // basic encryption provider
        Security.insertProviderAt(new BouncyCastleJsseProvider(),2); // tls
        Security.removeProvider("SunRsaSign");
        Security.removeProvider("SunJSSE");
    }

for your Runtime you can probably replicate the issue just like I have. It is important to fully change out the backing security implementation and also access Netty with JDK and not OpenSSL. Due to security concerns, I will not be able to share the full Dockerfile with you but please mention if there is anything in particular you are interested in.

Possible Solution

Support native readout of Netty Security Context (SSLContext) and thus usability of an adapted JSSE through the JDK Provider instead of OpenSSL. Unfortunately I am lacking the insight on where this is currently problematic or wether I am missing a configuration of some sort.

Context

I am trying to achieve full FIPS compliance through AWS and am currently failing at achieving full compliancy due to the Handshake error.

Your Environment

• AWS Java SDK version used: 2.15.34
  implementation("software.amazon.awssdk:s3:${awsSdkVersion}") api("software.amazon.awssdk:sqs:${awsSdkVersion}")

• JDK version used:
  openjdk 11.0.9 2020-10-20
  OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
  OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9+11, mixed mode)

• Operating System and version:
  Red Hat Enterprise Linux 7

• Additional Info:
  ARG BC_VERSION=1.0.2
  ARG BC_TLS_VERSION=1.0.10

[jakobmoellerdev]

After some more investigation, I have found that the AwsCrtAsyncHttpClient connects successfully, unfortunately this is only preview so I cannot really use it.

What would really help me here is being able to manually influence the SSLContext used (not just the Trust or Key Manager), as otherwise I have no way of actually influencing the Handshake. Also maybe it would help if I could understand why there is no cipher negotiation happening.

[jakobmoellerdev]

Okay so I found the Issue:

Bouncy Castle ships by default these Ciphers in FIPS Mode

0 = "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384"
1 = "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
2 = "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA"
3 = "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
4 = "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
5 = "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
6 = "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
7 = "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
8 = "TLS_RSA_WITH_AES_256_CBC_SHA256"
9 = "TLS_RSA_WITH_AES_128_CBC_SHA256"
10 = "TLS_RSA_WITH_AES_256_CBC_SHA"
11 = "TLS_RSA_WITH_AES_128_CBC_SHA"

The AWS Sdk uses the SslContextBuilder with these Ciphers:

0 = "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
1 = "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
2 = "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"
3 = "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
4 = "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
5 = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"

The reason for this is at instanciation in AwaitCloseChannelPoolMap it does

return SslContextBuilder.forClient()
                                    .sslProvider(sslProvider)
                                    .ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE)
                                    .trustManager(getTrustManager())
                                    .keyManager(getKeyManager())
                                    .build();

This does not line up, and to my knowledge is both not FIPS compliant as well as undocumented. Also we should not default to HTTP2 Ciphers if we are using HTTP 1.1 like this:

return NettyNioAsyncHttpClient.builder()
                .sslProvider(OpenSsl.isAvailable() ? SslProvider.OPENSSL : SslProvider.JDK)
                .protocol(Protocol.HTTP1_1)

The reason this is is not FIPS-compliant is that most FIPS-compliant JSSE Providers disable GCM Algorithms due to the Problem of an IV necessary that comes out of the TLS Context which cannot be influenced by the library. (For more details feel free to read into it by referencing FIPS140-2 IG A.5 and the correspondingBouncy Castle Documentation for FJA/JSSE on this)

This means in conclusion that either we find a way to overwrite Ciphers used by the SDK (and thus can reach compliancy without OpenSSL or a JSSE that has non-FIPS compliant GCM algorithms) or fall back to Ciphers specified in the JDK, e.g. through SSLContext.getDefault().getDefaultSSLParameters().getCipherSuites()

This is bad, as AWS is advertising FIPS compliance with their specific FIPS Endpoints, but the SDK does not support actually using a FIPS-compliant HTTP Call.

[jakobmoellerdev]

After drawing in the project and replacing the code above with

            SslContextBuilder sslContextBuilder = SslContextBuilder.forClient()
                    .sslProvider(sslProvider);

            if (sslProvider == SslProvider.JDK) {
                sslContextBuilder.ciphers(null, SupportedCipherSuiteFilter.INSTANCE);
            } else {
                sslContextBuilder.ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE);
            }

            return sslContextBuilder.trustManager(getTrustManager())
                                    .keyManager(getKeyManager())
                                    .build();

everything is running without problems 👍

[debora-ito]

@jakobmoellersap thank you for the very detailed and clear report, and possible solution. We will investigate the issue.

[cenedhryn]

@jakobmoellersap We'll add support for FIPS compliant algorithms to the Netty client and are tracking this issue internally.

[zoewangg]

Hi, @jakobmoellersap thank you for your patience. We've released a fix in 2.15.62. Could you try with the latest version?

[github-actions]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.