Unable to create native image for GRPC project

[auke-]

Building a native image for a Micronaut GRPC project fails due to the absence of SubstrateVM substitutions for Netty. When the Micronaut HTTP client dependency is added (which pulls in the substitutions) the building of the images fails due to the detection of a direct/mapped ByteBuffer in the image heap, which seems to be caused by the Netty GRPC shutdown handler.

The following steps reproduce the issue:

The project is created with the Micronaut 1.1.3 (but I've also tried 1.2.0 RC1) command line tool:

$ mn create-app micronaut-grpc-graal --build=maven --profile grpc --features=graal-native-image,java

Building that project fails due to the absence of the SubstrateVM substitutes for Netty:

$ ./mvnw clean package && ./docker-build.sh
...
Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of io.netty.util.internal.PlatformDependent0. Detailed failure reason(s): The field java.lang.Long.value, where the value produced by the array index scale computation is stored, is not static.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of io.netty.util.internal.PlatformDependent0. Add a RecomputeFieldValue.FieldOffset manual substitution for io.netty.util.internal.PlatformDependent0.ADDRESS_FIELD_OFFSET. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field.
Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of io.micronaut.caffeine.cache.UnsafeRefArrayAccess. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of io.netty.util.internal.CleanerJava6. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.ArrayIndexScale automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayIndexScale(Class) was detected in the static initializer of io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayIndexScale(Class) for the array index scale computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of io.netty.buffer.AbstractReferenceCountedByteBuf. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: class initialization of class io.netty.util.internal.logging.Log4JLogger failed with exception java.lang.NoClassDefFoundError: org/apache/log4j/Priority. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.util.internal.logging.Log4JLogger to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.ConscryptAlpnSslEngine failed with exception java.lang.NoClassDefFoundError: org/conscrypt/BufferAllocator. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.handler.ssl.ConscryptAlpnSslEngine to explicitly request delayed initialization of this class.
...

Micronaut supplies those substitutes with their Netty components, which are also needed when using the Micronaut rest client.
The micronaut-http-client dependency is added to the pom file:

<dependency>
  <groupId>io.micronaut</groupId>
  <artifactId>micronaut-http-client</artifactId>
  <scope>compile</scope>
</dependency>

This gives a different error when building the native image:

$ ./mvnw clean package && ./docker-build.sh
...
Warning: Aborting stand-alone image build. Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time. A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option -H:ClassInitialization=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: 	object io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf
	object io.netty.buffer.ReadOnlyByteBuf
	object io.netty.buffer.UnreleasableByteBuf
	method io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeContinuationFrames(ChannelHandlerContext, int, ByteBuf, int, Http2CodecUtil$SimpleChannelPromiseAggregator)
Call path from entry point to io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeContinuationFrames(ChannelHandlerContext, int, ByteBuf, int, Http2CodecUtil$SimpleChannelPromiseAggregator):
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeContinuationFrames(DefaultHttp2FrameWriter.java:555)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeadersInternal(DefaultHttp2FrameWriter.java:534)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeaders(DefaultHttp2FrameWriter.java:266)
	at io.netty.handler.codec.http2.Http2OutboundFrameLogger.writeHeaders(Http2OutboundFrameLogger.java:60)
	at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53)
	at io.grpc.netty.NettyServerHandler$WriteMonitoringFrameWriter.writeHeaders(NettyServerHandler.java:958)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:205)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:146)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.handleServerHeaderDecodeSizeError(Http2ConnectionHandler.java:720)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.onStreamError(Http2ConnectionHandler.java:696)
	at io.grpc.netty.NettyServerHandler.onStreamError(NettyServerHandler.java:506)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:610)
	at io.grpc.netty.NettyServerHandler$GracefulShutdown.secondGoAwayAndClose(NettyServerHandler.java:921)
	at io.grpc.netty.NettyServerHandler$GracefulShutdown$1.run(NettyServerHandler.java:885)
	at com.oracle.svm.core.jdk.RuntimeSupport.executeHooks(RuntimeSupport.java:144)
	at com.oracle.svm.core.jdk.RuntimeSupport.executeStartupHooks(RuntimeSupport.java:89)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:145)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

Warning: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception

It seems like it tries to access a ByteBuffer when the shutdown handler is being invoked.

The GraalVM native image support in Micronaut works great for REST services, I would expect the same for GRPC services.

I've created a sample repo to reproduce the issue, see https://github.com/auke-/micronaut-grpc-graal.

[graemerocher]

so it is probably just missing the Graal metadata for the GRPC module. We will have to play with it and see if we can get it working. Thanks for the report

[auke-]

This Netty change might be helpful: netty/netty@f1495e1#diff-5cbe3026d0fe94bd7b8480f692edfb17R15

Args = --rerun-class-initialization-at-runtime=io.netty.handler.codec.http2.Http2CodecUtil \
       --delay-class-initialization-to-runtime=io.netty.handler.codec.http2.DefaultHttp2FrameWriter

[graemerocher]

It would be helpful if there weren't other breaking native image related changes in Netty. See micronaut-projects/micronaut-core#1802

[auke-]

Currently Micronaut uses Netty version 4.1.30.Final. The latest version (4.1.36.Final) contains
substitutes for GraalVM, which would make the substitutes in Micronaut obsolete. Unfortunately
Micronaut isn't compatible with Netty 4.1.36.Final. However, the substitutes from this version
can be applied to the current version of Micronaut, especially the two following lines:

  --rerun-class-initialization-at-runtime=io.netty.handler.codec.http2.Http2CodecUtil
  --initialize-at-run-time=io.netty.handler.codec.http2.DefaultHttp2FrameWriter

Those lines are taken from the following Netty change:
netty/netty@f1495e1#diff-5cbe3026d0fe94bd7b8480f692edfb17R15

When those options are applied to the sample project (by adding them to native-image.properties)
the project builds correctly in both the JVM and as a GraalVM native image. Unfortunately the
application doesn't work because the example project doesn't implement and exposes the example
service. We need to implement the service, e.g. like this:

@Singleton
public class ExampleEndpoint extends ExampleServiceGrpc.ExampleServiceImplBase {

  @Override
  public void send(ExampleRequest request,
      StreamObserver<ExampleReply> responseObserver) {
    ExampleReply reply = ExampleReply.newBuilder()
        .setMessage("Hi " + request.getName())
        .build();
    responseObserver.onNext(reply);
    responseObserver.onCompleted();
  }

}

With the service enabled the native compilation fails again with the following messages:

$ ./mvnw clean package && ./docker-build.sh
...
Warning: RecomputeFieldValue.ArrayBaseOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.arrayBaseOffset(Class) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): Could not determine the field where the value produced by the call to sun.misc.Unsafe.arrayBaseOffset(Class) for the array base offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
...
Error: Class that is marked for delaying initialization to run time got initialized during image building: com.google.protobuf.ExtensionRegistry. Try marking this class for build-time initialization with --initialize-at-build-time=com.google.protobuf.ExtensionRegistry
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

It seems like com.google.protobuf.UnsafeUtil and com.google.protobuf.ExtensionRegistry needs some configuration in order to work with GraalVM.

I've applied those changes to my sample repo: https://github.com/auke-/micronaut-grpc-graal

[ilopmar]

@auke- One comment. Please keep in mind that --rerun-class-initialization-at-runtime flag has been deprecated in Graal 19.0.0, so you should use --initialize-at-run-time and --initialize-at-build-time.

Regarding the error, you can try the suggested fix: --initialize-at-build-time=com.google.protobuf.ExtensionRegistry,com.google.protobuf.UnsafeUtil. Or, if there are more classes, try also --initialize-at-build-time=com.google.protobuf.

[auke-]

The last suggestion seems to work, i've added the following options:

       --rerun-class-initialization-at-runtime=io.netty.handler.codec.http2.Http2CodecUtil
       --initialize-at-run-time=io.netty.handler.codec.http2.DefaultHttp2FrameWriter
       --initialize-at-build-time=com.google.protobuf

This still generates two warnings, but the service seems to work:

Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 

I can't find a working alternative for --rerun-class-initialization-at-runtime.

[graemerocher]

Nice. To fix those errors I think there need to be some re-computes defined similar to https://github.com/micronaut-projects/micronaut-core/blob/master/http-netty/src/main/java/io/micronaut/http/netty/graal/MicronautSubstitutions.java#L56

For the usages of Unsafe.

Other than that the next step would be to include these flags directly in the metadata for the micronaut GRPC project, then each user would not need to do what you had to do.

[graemerocher]

@auke- with these flags with GraalVM 19.1 and Micronaut master snapshot I get

Warning: Using a deprecated option --rerun-class-initialization-at-runtime. Currently there is no replacement for this option. Try using --initialize-at-run-time or use the non-API option -H:ClassInitialization directly.
[grpc-example:57500]    classlist:   4,557.99 ms
[grpc-example:57500]        setup:     106.74 ms
Error: Incompatible change of initialization policy for io.netty.handler.codec.http2.Http2CodecUtil: trying to change RUN_TIME from the command line to RERUN from the command line
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

I guess there are still issues with the flags being fed to native image

[auke-]

I've updated the sample project. It works with Micronaut 1.1.3. I have also updated the GraalVM version from 19.0.0 to 19.0.2 and 19.1.0. All versions seem to work with the same options:

  --rerun-class-initialization-at-runtime=io.netty.handler.codec.http2.Http2CodecUtil
  --initialize-at-run-time=io.netty.handler.codec.http2.DefaultHttp2FrameWriter
  --initialize-at-build-time=com.google.protobuf

I haven't tried 1.1.4, but when I use Micronaut 1.2.0-RC2 the native image compilation fails with the same error you got:

Error: Incompatible change of initialization policy for io.netty.handler.codec.http2.Http2CodecUtil: trying to change RUN_TIME from the command line to BUILD_TIME from the command line
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

When I remove the options for the Netty packages from the native-image.properties file (and leave the option for the protobuf package) the options look like this:

 --initialize-at-build-time=com.google.protobuf

Now the compilation fails with the following error:

Error: Class that is marked for delaying initialization to run time got initialized during image building: io.netty.handler.codec.http2.Http2CodecUtil. Try marking this class for build-time initialization with --initialize-at-build-time=io.netty.handler.codec.http2.Http2CodecUtil
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

This is caused by the Graal options provided by Netty 4.1.37. When the option for Http2CodecUtil in
META-INF/native-image/io.netty/codec-http2/native-image.properties is changed to --rerun-class-initialization-at-runtime=io.netty.handler.codec.http2.Http2CodecUtil in the shaded jar the compilation succeeds again and the application is working with Micronaut 1.2.0-RC2 and Netty 4.1.37.

It looks like option --initialize-at-run-time=io.netty.handler.codec.http2.Http2CodecUtil provided by Netty 4.1.37 doesn't work, or at least doesn't work for Micronaut and GRPC.

[graemerocher]

Bummer, one of the problems of having these now embedded in the Netty JAR. I wonder why --initialize-at-run-time was added for that class

[graemerocher]

I have asked why this flag was added netty/netty@3eff1db#r34249173

[auke-]

It was added in netty/netty#9118 in an attempt to remove deprecated flags.

[vjovanov]

The error says what is the problem. You are trying to define two conflicting policies for the same class (io.netty.handler.codec.http2.Http2CodecUtil). Netty explicitly says that this class must be delayed so you can no override that decision.

Also, rerun is not really in the API anymore so I would advise against using it.

[graemerocher]

@vjovanov ok great, but without that flag the following error occurs so it seems like a catch 22 situation:

Error: Class that is marked for delaying initialization to run time got initialized during image building: io.netty.handler.codec.http2.Http2CodecUtil. Try marking this class for build-time initialization with --initialize-at-build-time=io.netty.handler.codec.http2.Http2CodecUtil
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

If one specifies --initialize-at-build-time=io.netty.handler.codec.http2.Http2CodecUtil then you get the conflicting policy error. If one doesn't you get the Class that is marked for delaying initialization to run time got initialized during image building error, so how does one resolve this?

[vjovanov]

You need to put a breakpoint in io.netty.handler.codec.http2.Http2CodecUtil.<clinit> and see who initialized it. Then, the culprit should also be delayed.

I am working on a framework to do this for you, but ATM you need to go with the breakpoint.

[graemerocher]

so would that involve attaching a debugger to the native image tool

[vjovanov]

Yes, you use --debug-attach.

[pete-woods]

FWIW, I'm still seeing the same issue with a clean generated micronaut project:

$ mn create-app com.example.grpc --profile grpc --lang java --build gradle --features=graal-native-image
| Generating Java project...
| Application created at /..../grpc
$ cd grpc/
$ ./gradlew assemble

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.5/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5s
13 actionable tasks: 13 executed
$ ./docker-build.sh 
Sending build context to Docker daemon  93.23MB
Step 1/9 : FROM oracle/graalvm-ce:19.0.0 as graalvm
 ---> d413331a996d
Step 2/9 : COPY . /home/app/grpc
 ---> 455939627924
Step 3/9 : WORKDIR /home/app/grpc
 ---> Running in ebb99c1be951
Removing intermediate container ebb99c1be951
 ---> aef172f85348
Step 4/9 : RUN gu install native-image
 ---> Running in 70f00c145b89
Downloading: Component catalog from www.graalvm.org
Processing component archive: Native Image
Downloading: Component native-image: Native Image  from github.com
Installing new component: Native Image licence files (org.graalvm.native-image, version 19.0.0)
Refreshed alternative links in /usr/bin/
Removing intermediate container 70f00c145b89
 ---> fc77c1f91794
Step 5/9 : RUN native-image --no-server -cp build/libs/grpc-*-all.jar
 ---> Running in b88cea353e5b
[grpc:6]    classlist:   8,517.31 ms
[grpc:6]        (cap):     756.44 ms
[grpc:6]        setup:   2,056.52 ms
Warning: class initialization of class io.netty.handler.ssl.ReferenceCountedOpenSslContext failed with exception java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSLPrivateKeyMethod. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslContext to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.JdkNpnApplicationProtocolNegotiator failed with exception java.lang.NoClassDefFoundError: org/eclipse/jetty/npn/NextProtoNego$Provider. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.handler.ssl.JdkNpnApplicationProtocolNegotiator to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.ReferenceCountedOpenSslEngine failed with exception java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSL. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslEngine to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.util.internal.logging.Log4JLogger failed with exception java.lang.NoClassDefFoundError: org/apache/log4j/Priority. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.util.internal.logging.Log4JLogger to explicitly request delayed initialization of this class.
Warning: class initialization of class io.netty.handler.ssl.ConscryptAlpnSslEngine failed with exception java.lang.NoClassDefFoundError: org/conscrypt/BufferAllocator. This class will be initialized at run time because either option --report-unsupported-elements-at-runtime or option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.handler.ssl.ConscryptAlpnSslEngine to explicitly request delayed initialization of this class.
[grpc:6]     analysis:  43,472.81 ms
Error: Class that is marked for delaying initialization to run time got initialized during image building: io.netty.handler.codec.http2.Http2CodecUtil. Try marking this class for build-time initialization with --initialize-at-build-time=io.netty.handler.codec.http2.Http2CodecUtil
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1
The command '/bin/sh -c native-image --no-server -cp build/libs/grpc-*-all.jar' returned a non-zero code: 1

[pete-woods]

Adding:

       --initialize-at-run-time=io.netty.handler.codec.http2.Http2ClientUpgradeCodec \
       --initialize-at-run-time=com.google.protobuf.ExtensionRegistryFactory

to the native-image.properties file did the trick (following @vjovanov 's advice to attach the debugger).

[pete-woods]

Created a PR for netty: netty/netty#9593

[graemerocher]

@pete-woods Thanks for looking into it. We will see if we can get updated metadata published for this project

[pete-woods]

Cool. I couldn’t figure out where the protobuf metatdata came from

[graemerocher]

I have pushed a change that includes the non-Netty metadata in the grpc and protobuf modules (which are typically used separately).

[pete-woods]

There's another breakage in the newer Netty here that I've pushed a PR for: netty/netty#9621

[everflux]

The PR is merged since netty 4.1.43 it seems which is the netty dependency resolved as of micronaut 1.2.8 and at least for a small sample grpc app it works with graalvm-ce:19.3.0.2 native image. I assume this issue is therefore resolved.

[graemerocher]

Thanks for verifying