cmdLineTester_jvmtitests_hcr_OSRG_nongold failures with JITServer on x86_64 and JDK11

[cjjdespres]

The tests cmdLineTester_jvmtitests_hcr_OSRG_nongold_0 and cmdLineTester_jvmtitests_hcr_OSRG_nongold_2 have started failing in the JITServer nightly tests. Both fail in the rtc003 subtest, and only on X with JDK11, from what I can see. The earliest failure is this May 8 run. Console log for one of the failures in that run:

03:32:06  Eclipse OpenJ9 VM (build master-bfaeffa, JRE 11 Linux amd64-64-Bit Compressed References 20240508_1838 (JIT enabled, AOT enabled)
03:32:06  OpenJ9   - bfaeffa
03:32:06  OMR      - 1bcf969
03:32:06  JCL      - b8f6121 based on jdk-11.0.24+1)

        Testing: rtc003
        Test start time: 2024/05/08 01:12:04 Pacific Standard Time
        Running command: "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/bin/java" -XX:+UseJITServer -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation  -Xdump    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED -agentlib:jvmtitest=test:rtc003 -cp "/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar:/home/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_linux_jit_Personal_testList_0/../../testDependency/lib/asm-all.jar" com.ibm.jvmti.tests.util.TestRunner
        Time spent starting: 6 milliseconds
        Time spent executing: 1422 milliseconds
        Test result: FAILED
        Output from test:
         [OUT] *** Testing [1/1]:	testRedefineImplementedInterface
         [ERR] Exception in thread "main" java.lang.NoClassDefFoundError: com.ibm.oti.util.Msg (initialization failure)
         [ERR] 	at java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:160)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:865)
         [ERR] 	at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:859)
         [ERR] 	at java.base/java.lang.Thread.uncaughtException(Thread.java:1363)
         [ERR] Caused by: java.lang.ClassCastException: K0340
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.open(SystemModuleFinders.java:448)
         [ERR] 	at java.base/jdk.internal.loader.BuiltinClassLoader.findResourceAsStream(BuiltinClassLoader.java:434)
         [ERR] 	at java.base/jdk.internal.loader.BootLoader.findResourceAsStream(BootLoader.java:146)
         [ERR] 	at java.base/java.lang.Module.getResourceAsStream(Module.java:1590)
         [ERR] 	at java.base/com.ibm.oti.vm.MsgHelp.loadMessages(MsgHelp.java:185)
         [ERR] 	at java.base/com.ibm.oti.util.PriviAction.run(PriviAction.java:172)
         [ERR] 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:690)
         [ERR] 	at java.base/com.ibm.oti.util.Msg.<clinit>(Msg.java:63)
         [ERR] 	at java.base/java.lang.ClassCastException.<init>(ClassCastException.java:71)
         [ERR] 	at java.base/java.lang.invoke.HandleCache.getStaticCache(HandleCache.java:124)
         [ERR] 	at java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:644)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.sendResolveMethodHandle(MethodHandleResolver.java:313)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
         [ERR] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:216)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:49)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003$TestFunction.apply(rtc003.java:47)
         [ERR] 	at java.base/java.util.Map.computeIfAbsent(Map.java:1003)
         [ERR] 	at com.ibm.jvmti.tests.retransformClasses.rtc003.testRedefineImplementedInterface(rtc003.java:38)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         [ERR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         [ERR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         [ERR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:572)
         [ERR] 	at com.ibm.jvmti.tests.util.TestCase.run(TestCase.java:215)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:68)
         [ERR] 	at com.ibm.jvmti.tests.util.TestSuite.run(TestSuite.java:79)
         [ERR] 	at com.ibm.jvmti.tests.util.TestRunner.main(TestRunner.java:60)
        >> Success condition was not found: [Return code: 0]

A 25x grinder without JITServer did not result in any failures, and this is reproducible consistently locally with JITServer enabled.

These are the differences between failing May 8 build and the successful May 7 build:

• b05b2c1...bfaeffa
• eclipse/omr@acd6f7e...1bcf969
• adoptium/aqa-tests@348e551...b2a0d5a

I tested a build based on b05b2c1 and eclipse/omr@1bcf969 (so excluding bfaeffa in particular) and it did not fail, based on 3 test iterations. I tested a build based on bfaeffa and eclipse/omr@acd6f7e, and 3 test iterations both failed. So, it may be #19355 that is causing the failure, or at least causing it to appear consistently.

[cjjdespres]

Attn @mpirvu. I'll check to see if it's only the first commit of that PR, or if it needs both.

[cjjdespres]

I get no failures if I start at bfaeffa and revert just 8dd5a8c, so both seem to be required.

[mpirvu]

@cjjdespres Could you please check if the test fails when we disable the disclaim with command line options: -Xjit:disableIprofilerDataDisclaiming

[cjjdespres]

The test still fails. I used EXTRA_OPTIONS="-Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation,disableIprofilerDataDisclaiming" because this test sets those other -Xjit options.

[mpirvu]

Just to double check: this test does not fail without JITServer, correct?

[cjjdespres]

That's right, or at least a 25x grinder without JITServer had 0 failures and with JITServer it fails every time.

[cjjdespres]

These tests were failing consistently up to the May 27 nightly run. They started passing on the May 28 run and have not failed since.

The difference between the May 27 and May 28 runs in openj9 is ce04759...4ff10c9. Between those commits the PR #19545 was merged, which fixed an error introduced in #19355 (that PR being the apparent cause of these failures). So, that former PR may have also fixed this issue as well.

I'm still unsure how those PRs could have introduced or fixed the problem, but I think this issue can be closed in any case.