EasyMock 5.0.1 OOM on large project

[wroth]

We have a very large Java 11 project, using easymock 4.3, that has tons of illegal accesses. (I.e. many many tests fail when we enable --illegal-access=deny.) Everything works fine right now, we're just looking ahead to the future (i.e. Java 17, when 'deny' becomes the default.)

Building with easymock 5.0.1, the build (gradle 5) OOMs after running the first ~11,000 tests. (There are 20,000+). Watching the CPU, looks like a clear sign of thrashing/memory leak, as the # of tests run increases. Telling our gradle build to fork a new JVM every 100 test classes makes everything work OK, but slower.

I grabbed a heap dump shortly before the OOM -- all 667MB of it -- and I see 1.5M instances of "java.lang.reflect.Method", which seems unusual, to say the least.

I don't know how to diagnose this further, but it certainly smells like a memory leak in/around easymock 5. I can post the heap dump file somewhere public if that is useful. Or happy to perform any other experiments as suggested.

Sorry to be so vague, but I wanted to offer everything we know at the moment. The Gradle "forkEvery" option will work for us, but wanted to post the issue in case it is useful.

[henri-tremblay]

Thanks for the feedback. I will try to reproduce. But can you please look at the GC root of the Method instance?

[wcroth55]

Thanks. You can download the heap dump (670MB, yikes) from https://mivoter.org/core. Attempting to attach a screenshot of the gc root display. It's been a long time since I used jvisualvm, so not sure how much analysis I'm capable of.

[beatngu13]

We can confirm this issue with EasyMock 5.0.1. Heap shows many instances of java.lang.reflect.Method, then tests start failing with:

java.lang.IllegalStateException: java.lang.OutOfMemoryError: Metaspace
	at org.easymock.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$Enabled.defineClass(ClassInjector.java:2051)
	at org.easymock.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe.injectRaw(ClassInjector.java:1823)
	at org.easymock.bytebuddy.dynamic.loading.ClassInjector$AbstractBase.inject(ClassInjector.java:118)
	at org.easymock.bytebuddy.dynamic.loading.ClassLoadingStrategy$ForUnsafeInjection.load(ClassLoadingStrategy.java:595)
	at org.easymock.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:101)
	at org.easymock.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:6317)
	at org.easymock.internal.ClassProxyFactory.createProxy(ClassProxyFactory.java:121)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:108)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:86)
	at org.easymock.EasyMockSupport.mock(EasyMockSupport.java:149)
	at org.easymock.EasyMockSupport.createMock(EasyMockSupport.java:303)
	at org.easymock.internal.Injector.createMocksForAnnotations(Injector.java:160)
	at org.easymock.internal.Injector.injectMocks(Injector.java:67)
	at org.easymock.EasyMockSupport.injectMocks(EasyMockSupport.java:605)
	[...]

Feel free to let us know if there is any additional information we can provide, that might help.

[nabeygun]

I am unsure if it is related but this article on ByteBuddy references a possible memory leak if the proxied classes are cached:
http://mydailyjava.blogspot.com/2018/04/jdk-11-and-proxies-in-world-past.html

Unfortunately, caching classes in Java brings some caveats. If a proxy is created, it does of course subclass the class it proxies what makes this base class ineligible for garbage collection. Therefore, if the proxy class was referenced strongly, the key would also be referenced strongly. This would render the cache useless and open for memory leaks. Therefore, the proxy class must be referenced softly or weakly what is specified by the constructor argument. In the future, this problem might be resolved if Java introduced ephemerons as a reference type. At the same time, if garbage collection of proxy classes is not an issue, a ConcurrentMap can be used to compute a value on absence.

[henri-tremblay]

I have started to look at it and I tried the TypeCache from ByteBuddy to fix it. However, there's something that sticks between mocks extending the same class. So I need to look a bit more into it. I'll push a branch if someone wants to have a look

[henri-tremblay]

I have created a branch named typecache showing the issue.

[john9x]

@henri-tremblay What's wrong with my PR for type caching ? All tests (including new tests in ClassProxyFactoryTest from typecache branch) passes fine with it.

[henri-tremblay]

It's fine now. I think I can merge and release. I will find the time during Christmas.

[wroth]

@henri-tremblay Thanks so much!

Will this be in 5.0.2? Or is there a public checkout available of this fix? I'd love to try it on our 22,000+ test base, and report back.

Also, do you have a coffee/pizza/whatever fund? :-)

[henri-tremblay]

It's EasyMock 5.1.0. Let me see for the pizza :-)

[beatngu13]

We updated to 5.1.0 and on the build server (Jenkins on Linux) with:

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/maven-3.8
Java version: 1.8.0_332, vendor: Temurin, runtime: /opt/jenkins/tools/hudson.model.JDK/openjdk8/jdk8u332-b09/jr

Everything was fine.

However, on local Windows machines with:

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: C:\LProg\Ldev\apache-maven
Java version: 1.8.0_352, vendor: Red Hat, Inc., runtime: C:\Program Files\OpenJDK\1.8.0\jre

We (still) get:

Unexpected IOException with stream: Jan 10, 2023 12:53:52 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNUNG: TestEngine with ID 'junit-jupiter' failed to execute tests
java.lang.OutOfMemoryError: Metaspace

The issue goes away when downgrading to EasyMock 4.3.

Is anyone experiencing similar issues?

[wroth]

Our large (22K tests) codebase "worked" under EasyMock 5.0.1, but threw OOMS as noted earlier.

I just tried 5.1.0 today, and now I have many tests failing, e.g.
java.lang.RuntimeException at LionFullTextFormatterTest.java:58
Caused by: java.lang.IllegalAccessException at LionFullTextFormatterTest.java:58
Caused by: java.lang.LinkageError at LionFullTextFormatterTest.java:58

... and that line (58) is just calling createMock with a controller (from EasyMock.createControl()).

[wroth]

@henri-tremblay Note that 5.1.0 is still failing. I suggest this is not "closed" (unless there's a different ticket for 5.1.0).

This is Java 11 (Corretto) with Gradle 5, on both Windows 10 and Oracle Linux 8.