Upgrades Netty to 4.1.70.Final

[ljnelson]

Upgrades Netty to 4.1.68.Final and adds apparently necessary initialize-at-runtime statements to the native image properties in the webserver project.

Signed-off-by: Laird Nelson laird.nelson@oracle.com

[ljnelson]

@barchetta @tomas-langer Joe asked for a volunteer to do this. Specifically the mp-1 test was failing with just an "ordinary" Netty upgrade, so I ran it and added relevant initialize-at-runtime lines. I did so by taking the instructions that GraalVM spat out and incorporated them rather blindly into the native-image.properties file. This seemed to make the mp-1 test pass but I stress that this is definitely not an area of expertise so drop some eyes on this please and I'll undraft it if things look OK.

[tomas-langer]

The native-image.properties seems to be updated fine.

[tomas-langer]

The test failure can be reproduced locally, so it is most likely caused by the upgrade.

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.314 s <<< FAILURE! - in io.helidon.grpc.server.SslIT
[ERROR] io.helidon.grpc.server.SslIT  Time elapsed: 0.314 s  <<< ERROR!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: io/netty/internal/tcnative/AsyncSSLPrivateKeyMethod

[ljnelson]

OK; looks like pom.xml (not dependencies/pom.xml for some reason?) needs to be updated to also contain a higher version of netty-tcnative-boringssl-static. I'll make that change.

[ljnelson]

This now results in a different error:

[ERROR] io.helidon.grpc.server.SslIT.shouldConnectWithoutClientCertsFor1Way  Time elapsed: 0.063 s  <<< ERROR!
io.grpc.StatusRuntimeException: UNAVAILABLE: GOAWAY shut down transport. HTTP/2 error code: INTERNAL_ERROR, debug data: io.netty.util.IllegalReferenceCountException: refCnt: 0
	at io.helidon.grpc.server.SslIT.shouldConnectWithoutClientCertsFor1Way(SslIT.java:129)

[ljnelson]

I'll commit the tcnative version update because it needs to be there, and then perhaps @aseovic can take a look at the grpc innards portion of this.

[barchetta]

FYI, in 4.1.65.Final they disabled tlsv1 by default. Maybe that impacts the test: https://netty.io/news/2021/05/19/4-1-65-Final.html

We should consider the compatibility impact of that on Helidon.

[ljnelson]

Rebased this PR and bumped Netty to 4.1.70.Final to see if that helps.

[ljnelson]

Also had to bump up the tcnative version to conform to https://github.com/netty/netty/blob/f49c94c83a5b2a2625913f57515c8dc6ab8c2639/pom.xml#L511

[ljnelson]

Keeping an eye on grpc/grpc-java#8605 which seems to be responsible.

[ljnelson]

Notes to myself. Root stack trace:

2021.11.16 15:56:56 INFO io.grpc.netty.NettyServerTransport.connections Thread[nioEventLoopGroup-7-3,5,main]: Transport failed
io.netty.handler.codec.DecoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
	at io.helidon.grpc.server.GrpcServerImpl$ContextAwareThreadFactory.lambda$newThread$0(GrpcServerImpl.java:472)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0
	at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454)
	at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1383)
	at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1379)
	at io.netty.buffer.AbstractByteBuf.getByte(AbstractByteBuf.java:355)
	at io.netty.buffer.AbstractByteBuf.getUnsignedByte(AbstractByteBuf.java:368)
	at io.netty.handler.ssl.SslUtils.getEncryptedPacketLength(SslUtils.java:280)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1210)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)

[duplexsystem]

I'd recommend updating io_uring to 0.0.10 along with this PR :) as io_uring bump it's netty dependency to 4.1.70 in 0.0.10. No code changes for that but probably best practice to update it as well.

[ljnelson]

I can make the failing test pass by effectively undoing netty/netty#11242. More investigation to come.

[ljnelson]

Filed: netty/netty-tcnative#680

[normanmaurer]

@ljnelson how can I reproduce this ? I did checkout your branch and run mvn clean package but it passes.

[barchetta]

@normanmaurer this should reproduce (using Laird's branch):

# java 17.0.1 2021-10-19 LTS
# Apache Maven 3.8.2
# From top of repo
mvn clean install -DskipTests
cd grpc/server
mvn clean install

[normanmaurer]

@barchetta no luck :(

[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ helidon-grpc-server ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 16 source files to /Users/norman/Documents/workspace/helidon/grpc/server/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ helidon-grpc-server ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.helidon.grpc.server.GrpcServiceTest
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.491 s - in io.helidon.grpc.server.GrpcServiceTest
[INFO] Running io.helidon.grpc.server.GrpcServerConfigurationTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.206 s - in io.helidon.grpc.server.GrpcServerConfigurationTest
[INFO] Running io.helidon.grpc.server.HealthServiceImplTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 s - in io.helidon.grpc.server.HealthServiceImplTest
[INFO] Running io.helidon.grpc.server.MethodDescriptorTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 s - in io.helidon.grpc.server.MethodDescriptorTest
[INFO] Running io.helidon.grpc.server.ServiceDescriptorTest
[INFO] Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 s - in io.helidon.grpc.server.ServiceDescriptorTest
[INFO] Running io.helidon.grpc.server.ContextSettingServerInterceptorTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 s - in io.helidon.grpc.server.ContextSettingServerInterceptorTest
[INFO] Running io.helidon.grpc.server.ConstantHealthCheckTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in io.helidon.grpc.server.ConstantHealthCheckTest
[INFO] Running io.helidon.grpc.server.BindableServiceImplTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in io.helidon.grpc.server.BindableServiceImplTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ helidon-grpc-server ---
[INFO] Building jar: /Users/norman/Documents/workspace/helidon/grpc/server/target/helidon-grpc-server-2.4.1-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.502 s
[INFO] Finished at: 2021-11-22T18:59:55+01:00
[INFO] ------------------------------------------------------------------------
➜  server git:(upgrade-netty-to-4168Final) ✗ echo $JAVA_HOME
/Users/norman/.sdkman/candidates/java/current
➜  server git:(upgrade-netty-to-4168Final) ✗ /Users/norman/.sdkman/candidates/java/current/bin/java -version
openjdk version "17" 2021-09-14 LTS
OpenJDK Runtime Environment Zulu17.28+13-CA (build 17+35-LTS)
OpenJDK 64-Bit Server VM Zulu17.28+13-CA (build 17+35-LTS, mixed mode, sharing)

git branch:

upgrade-netty-to-4168Final

[ljnelson]

I don't know anything about the Zulu OpenJDK distribution. Is it possible it causes some different SSL engine to be used in Netty? If this issue is too hard to use for a reproducer (none of us has had this problem), you may find that reproducing the same issue over in grpc-java land is easier; see grpc/grpc-java#8605. Of course over there you'll need to run on Java 8.

[ljnelson]

@normanmaurer Sorry, read everything too fast. package (and install) doesn't go "far" enough; you'll need to do at least mvn verify. The failing test is run by the maven-failsafe-plugin as an integration test during the verify phase.

[normanmaurer]

@ljnelson thanks a lot! This allows me to reproduce locally now. Will come back to you

[ljnelson]

Here is my (Netbeans) breakpoint that let me find the problem:

[ljnelson]

Finally, I debugged all this by running this Maven command:

JAVA_HOME=$(/usr/libexec/java_home -v17) /usr/local/mvn/bin/mvn verify -Dmaven.failsafe.debug -Dit.test='SslIT#shouldConnectWithClientCertsFor2WayUseConfig'

I mention this because the maven.failsafe.debug property turned out to be very convenient and together with the it.test property made for comparatively rapid turnarounds.

[normanmaurer]

I can confirm this fixes it:

diff --git a/grpc/server/src/main/java/io/helidon/grpc/server/GrpcServerImpl.java b/grpc/server/src/main/java/io/helidon/grpc/server/GrpcServerImpl.java
index 227f6f281..179014648 100644
--- a/grpc/server/src/main/java/io/helidon/grpc/server/GrpcServerImpl.java
+++ b/grpc/server/src/main/java/io/helidon/grpc/server/GrpcServerImpl.java
@@ -66,6 +66,7 @@ import io.netty.channel.nio.NioEventLoopGroup;
 import io.netty.channel.socket.nio.NioServerSocketChannel;
 import io.netty.handler.ssl.ClientAuth;
 import io.netty.handler.ssl.JdkSslContext;
+import io.netty.handler.ssl.OpenSslContextOption;
 import io.netty.handler.ssl.SslContext;
 import io.netty.handler.ssl.SslContextBuilder;
 import io.netty.handler.ssl.SslProvider;
@@ -414,7 +415,7 @@ public class GrpcServerImpl implements GrpcServer {
         }

         SslContextBuilder sslContextBuilder = SslContextBuilder.forServer(certResource.stream(), keyResource.stream())
-                .sslProvider(SslProvider.OPENSSL);
+                .sslProvider(SslProvider.OPENSSL).option(OpenSslContextOption.USE_TASKS, false);

         if (aX509Certificates.length > 0) {
             sslContextBuilder.trustManager(aX509Certificates)
diff --git a/grpc/server/src/test/java/io/helidon/grpc/server/SslIT.java b/grpc/server/src/test/java/io/helidon/grpc/server/SslIT.java
index ac6e8d2bf..a24f4f28b 100644
--- a/grpc/server/src/test/java/io/helidon/grpc/server/SslIT.java
+++ b/grpc/server/src/test/java/io/helidon/grpc/server/SslIT.java
@@ -39,6 +39,7 @@ import io.grpc.StatusRuntimeException;
 import io.grpc.netty.GrpcSslContexts;
 import io.grpc.netty.NegotiationType;
 import io.grpc.netty.NettyChannelBuilder;
+import io.netty.handler.ssl.OpenSslContextOption;
 import io.netty.handler.ssl.SslContext;
 import io.netty.handler.ssl.SslContextBuilder;
 import org.junit.AfterClass;
@@ -255,6 +256,7 @@ public class SslIT {
         if (clientCertChainFilePath != null && clientPrivateKeyFilePath != null) {
             builder.keyManager(clientCertChainFilePath.stream(), clientPrivateKeyFilePath.stream());
         }
+        builder.option(OpenSslContextOption.USE_TASKS, false);
         return builder.build();
     }

So definitely a netty bug.

[ljnelson]

Thanks, @normanmaurer; looks like your netty/netty#11854 will fix this.

[ljnelson]

Superseded by #3720