system.cpu.usage does not work in a native image

[wilkinsona]

Describe the bug

2023-01-05T15:29:04.211Z  WARN 97371 --- [  XNIO-1 task-2] i.m.c.instrument.internal.DefaultGauge   : Failed to apply the value function for the gauge 'system.cpu.usage'. Note that subsequent logs will be logged at debug level.

com.oracle.svm.core.jdk.UnsupportedFeatureError: Runtime reflection is not supported for public abstract double com.sun.management.OperatingSystemMXBean.getCpuLoad()
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:89) ~[na:na]
	at java.base@17.0.5/java.lang.reflect.Method.acquireMethodAccessor(Method.java:71) ~[actuator-undertow:na]
	at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:566) ~[actuator-undertow:na]
	at io.micrometer.core.instrument.binder.system.ProcessorMetrics.invoke(ProcessorMetrics.java:117) ~[actuator-undertow:1.10.2]
	at io.micrometer.core.instrument.binder.system.ProcessorMetrics.lambda$bindTo$0(ProcessorMetrics.java:104) ~[actuator-undertow:1.10.2]
	at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:53) ~[na:na]
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$5(PrometheusMeterRegistry.java:331) ~[actuator-undertow:1.10.2]
	at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:75) ~[na:na]
	at io.prometheus.client.Collector.collect(Collector.java:45) ~[actuator-undertow:na]
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204) ~[na:na]
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219) ~[na:na]
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152) ~[na:na]
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71) ~[na:na]
	at org.springframework.boot.actuate.metrics.export.prometheus.TextOutputFormat$1.write(TextOutputFormat.java:45) ~[na:na]
	at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape(PrometheusScrapeEndpoint.java:62) ~[actuator-undertow:3.0.0-SNAPSHOT]
	at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568) ~[actuator-undertow:na]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:281) ~[na:na]
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74) ~[na:na]
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) ~[actuator-undertow:3.0.0-SNAPSHOT]
	at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:317) ~[na:na]
	at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:423) ~[actuator-undertow:3.0.0-SNAPSHOT]
	at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568) ~[actuator-undertow:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207) ~[actuator-undertow:6.0.2]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1080) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:973) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1003) ~[actuator-undertow:6.0.2]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:895) ~[actuator-undertow:6.0.2]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[actuator-undertow:6.0.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:880) ~[actuator-undertow:6.0.2]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[actuator-undertow:6.0.0]
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[na:na]
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[na:na]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[actuator-undertow:6.0.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[actuator-undertow:6.0.2]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[na:na]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[actuator-undertow:6.0.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[actuator-undertow:6.0.2]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[na:na]
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109) ~[actuator-undertow:6.0.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[actuator-undertow:6.0.2]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[na:na]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[actuator-undertow:6.0.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[actuator-undertow:6.0.2]
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[na:na]
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[na:na]
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[na:na]
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) ~[na:na]
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[na:na]
	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) ~[na:na]
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) ~[na:na]
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[na:na]
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[na:na]
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[na:na]
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[na:na]
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[na:na]
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[actuator-undertow:2.3.0.Final]
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[na:na]
	at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) ~[na:na]
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[na:na]
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275) ~[na:na]
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134) ~[na:na]
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131) ~[na:na]
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[na:na]
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[na:na]
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255) ~[na:na]
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100) ~[na:na]
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:391) ~[na:na]
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) ~[na:na]
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) ~[na:na]
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513) ~[na:na]
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538) ~[na:na]
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) ~[na:na]
	at java.base@17.0.5/java.lang.Thread.run(Thread.java:833) ~[actuator-undertow:na]
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775) ~[actuator-undertow:na]
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203) ~[na:na]

Environment

• Micrometer version: 1.10
• Micrometer registry: N/A
• OS: macOS
• Java version: Graal 22.3.0

It looks like some more reflect-config entries are needed alongside the existing entries:

micrometer/micrometer-core/src/main/resources/META-INF/native-image/io.micrometer/micrometer-core/reflect-config.json

Lines 2 to 8 in 38003f5

	{
	"name":"com.sun.management.OperatingSystemMXBean",
	"methods":[
	{"name":"getProcessCpuLoad","parameterTypes":[] },
	{"name":"getSystemCpuLoad","parameterTypes":[] }
	]
	},

[jonatan-ivanov]

Thanks for the report.

I tried to reproduce it with a simple psvm native app and also with a simple Boot app (initializer example) using GraalVM 22.3.0 Java 17 CE but I was not able to.

Looking at the code and the config, the issue seems totally valid, I added it to reflect-config: 3c652d8.

Could you please help me to validate if this fixed the issue (latest snapshot should be fixed)?

Also, could you please help me to repro by answering these questions?

• Did you use GraalVM 22.3.0 Java 17 CE (so not 19 and not EE)?
• My Boot project (see link above) is similar enough to your right?
• Do you happen to have an M1 Mac? 😈

[matthenry87]

I also had to add this hint to eliminate this error in a native image.

Spring Boot 3.0.1

GraalVM 22.3.0 Java 17 CE
Java version info: '17.0.5+8-jvmci-22.3-b08'
C compiler: gcc (linux, x86_64, 9.4.0)
Garbage collector: Serial GC

[jonatan-ivanov]

@matthenry87 Thanks for the data point, for the record, here's mine:

Version info: 'GraalVM 22.3.0 Java 17 CE'
Java version info: '17.0.5+8-jvmci-22.3-b08'
C compiler: cc (apple, x86_64, 14.0.0)
Garbage collector: Serial GC

fyi: I added AOT support to our Boot 3 samples: micrometer-metrics/micrometer-samples#5
If I run the commands in the PR description, it works to me, probably won't work for you but if you force the latest snapshot, it should work.

[shakuzen]

I also was unable to reproduce the reported error on my M1 macbook using the micrometer-samples-boot3-web with Micrometer 1.10.2 and compiled to native with the same version of GraalVM Jonatan used (though arm64 arch).

Previously we removed thegetCpuLoad() reflection config because it caused issues for folks native compiling with a Java 11 based GraalVM version. From my testing, it appears that issue no longer happens on newer versions of GraalVM even ones targeting Java 11. I do see the following warning during the native compilation, but the image builds and runs.

Warning: Method com.sun.management.OperatingSystemMXBean.getCpuLoad() not found.

Could someone who is able to reproduce the reported issue here check if it is still present in the latest 1.10.3-SNAPSHOT version with the reflection config added for getCpuLoad()? Or alternatively give us steps so we can reproduce the error ourselves?

[wilkinsona]

--allow-incomplete-classpath is enabled by default and has been since early 2022, IIRC. I suspect this is why you haven't seen any problems in your testing with Java 11. You may see it if you use --link-at-build-time to switch back to the old behavior.

Apologies for the lack of repro details in the opening description. I saw the problem in a Boot app using Prometheus. The exception was thrown upon making a request to the prometheus endpoint. I'll verify that I can reproduce the problem and that it's resolved by 1.10.3-SNAPSHOT.

[wilkinsona]

I've reproduced the problem and now have a better understanding of why it happens and why you may not have been able to reproduce.

The problem only occurs when something else has made com.sun.management.OperatingSystemMXBean queryable
via reflection. If method invocation has not been enabled, the failure reported in the opening comment occurs. This will happen when using Undertow, for example, as its reachability metadata (incorrectly) allows the public methods of OperatingSystemMXBean to be queried. oracle/graalvm-reachability-metadata#161 should fix this particular cause.

I've tested with Micrometer 1.10.3-SNAPSHOT and the problem no longer occurs: system_cpu_usage appears in the Prometheus endpoint's response with a numeric value rather than NaN.

[jonatan-ivanov]

@wilkinsona @shakuzen Thank you!

Just for the sake of referencing this:
fix 3c652d8

[shakuzen]

We discussed this some more and we will be rolling back the changes in the 1.8.x and 1.9.x lines so we aren't potentially breaking compatibility for folks in those lines. We will keep the fix in the 1.10.x line for the upcoming 1.10.3 patch, as we feel the behavior we shipped in 1.10 was wrong and should be corrected in a patch release. It has a lower chance of causing issues as we expect users to be on a more recent version of GraalVM with Micrometer 1.10. We can reconsider what to do if there remain issues for users.