ReferenceCountedOpenSslEngine on jdk without TLSv1.3 should set SSL_OP_NO_TLSv1_3

[oskarwiksten]

Expected behavior

Using an older jdk such as 1.8.0_172 without support for TLSv1.3 should set the SSL_OP_NO_TLSv1_3 bit in openssl options.

Actual behavior

The SSL_OP_NO_TLSv1_3-bit ("do not use TLSv1.3"-bit) is only set if the list of protocols is explicitly specified, even if the jdk lacks support for TLSv1.3.

The consequence of this bit not being set is that tls handshakes for a server on such a jdk will use TLSv1.3, eventhough it is not fully supported.

Steps to reproduce

1. Run on a jdk that does not have support for TLSv1.3, or hardcode io.netty.handler.ssl.SslUtils::isTLSv13SupportedByJDK to always return false
2. Do not set "allowed protocols"
3. Create ReferenceCountedOpenSslEngine
4. Examine the SSL_OP_NO_TLSv1_3-bit in openssl options
5. Expected behaviour is that the bit should be set, since the jdk does not have support for TLSv1.3

Minimal yet complete reproducer code (or URL to code)

public static void main(String[] args) {
	try {
		final SslProvider openssl = SslProvider.OPENSSL_REFCNT;
		System.out.println("java.version                 : " + System.getProperty("java.vm.name") + " " + System.getProperty("java.version"));
		System.out.println("OpenSsl.versionString        : " + OpenSsl.versionString());
		System.out.println("SslProvider.isTlsv13Supported: " + SslProvider.isTlsv13Supported(openssl));
		final SelfSignedCertificate cert = new SelfSignedCertificate();
		final ReferenceCountedOpenSslEngine sslEngine = (ReferenceCountedOpenSslEngine) SslContextBuilder.forServer(cert.key(), cert.cert())
				.sslProvider(openssl)
				.build()
				.newEngine(UnpooledByteBufAllocator.DEFAULT);
		System.out.println("sslEngine.getEnabledProtocols: " + Arrays.asList(sslEngine.getEnabledProtocols()));
		System.out.println("'Do not use TLSv1.3' before  : " + hasSetOptionsBit_DoNotUseTlsv13(sslEngine));
		sslEngine.setEnabledProtocols(sslEngine.getEnabledProtocols());
		System.out.println("'Do not use TLSv1.3' after   : " + hasSetOptionsBit_DoNotUseTlsv13(sslEngine));
	} catch (Exception e) {
		e.printStackTrace();
	}
}

private static boolean hasSetOptionsBit_DoNotUseTlsv13(ReferenceCountedOpenSslEngine sslEngine) {
	return (SSL.getOptions(sslEngine.sslPointer()) & SSL.SSL_OP_NO_TLSv1_3) != 0;
}

Output from above code:

java.version                 : Java HotSpot(TM) 64-Bit Server VM 1.8.0_172
OpenSsl.versionString        : BoringSSL
SslProvider.isTlsv13Supported: false
sslEngine.getEnabledProtocols: [SSLv2Hello, TLSv1.2]
'Do not use TLSv1.3' before  : false
'Do not use TLSv1.3' after   : true

• Above code reproduces the problem on Oracle jdk 1.8.0_172, which does not have support for TLSv1.3
• To reproduce on a later jdk, it might be easiest to temporarily hardcode io.netty.handler.ssl.SslUtils::isTLSv13SupportedByJDK to always return false
• Expected behaviour is that 'Do not use TLSv1.3' before should also print true.

Netty version

• io.netty:netty-handler: 4.1.84.Final
• io.netty:netty-tcnative-boringssl-static: 2.0.54.Final

JVM version (e.g. java -version)

java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

OS version (e.g. uname -a)

Linux hostname 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Workarounds

If either of the following things is done, the issue does not happen:

• Using a newer jdk that has support for TLSv1.3
• Using Respect jdk.tls.client.protocols and jdk.tls.server.protocols #12797 and setting -Djdk.tls.server.protocols=TLSv1.2
• Explicitly setting the list of protocols via SslContextBuilder::protocols or ReferenceCountedOpenSslEngine::setEnabledProtocols
• Calling sslEngine.setEnabledProtocols(sslEngine.getEnabledProtocols()) as indicated in above reproducer-code

Now, using such an old jdk might be a niche use-case and possibly something that is not supported. But the fact that calling e.setEnabledProtocols(e.getEnabledProtocols()) has an effect on the options-bits seems unexpected.

[chrisvest]

The ReferenceCountedOpenSslEngine implements SSLEngine by using OpenSSL via JNI. Whether or not the JDK implementation of SSLEngine supports TLSv1.3 should not impact what you can do with OpenSSL. In fact, using OpenSSL is the work-around for lack of TLSv1.3 support in older JDKs, no?

[oskarwiksten]

Whether or not the JDK implementation of SSLEngine supports TLSv1.3 should not impact what you can do with OpenSSL

There is code in place that explicitly checks for whether or not the JDK supports TLSv1.3 , even when using OpenSSL. From #11604 . As seen in the reproducer code above, that check seems to correctly disable TLSv1.3 from getEnabledProtocols() when running with OpenSSL on a JDK that does not support it. However, it does not seem to disable TLSv1.3 in the OpenSSL options-bits.

[normanmaurer]

@oskarwiksten PTAL #13066 ... I am a bit surprised that it was not enough what we did here tho:
https://github.com/netty/netty/blob/4.1/handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslContext.java#L285

[oskarwiksten]

PTAL #13066

Thank you @normanmaurer . This resolves the issue we were seeing. Have also verified the fix on the application we had the original problem with, running on JRE 1.8.0_172. TLS handshakes now work as expected when running on such an old JRE.