@Async with cglib based proxy causes memory leak in heap [SPR-11275]

[spring-projects-issues]

Lukasz Rozek opened SPR-11275 and commented

running mvn clean package && mvn exec:exec
against repro project that can be found here
https://github.com/lrozek/spring-leak
or https://github.com/lrozek/spring-leak/archive/master.zip
or git clone https://github.com/lrozek/spring-leak.git
causes java.lang.OutOfMemoryError: GC overhead limit exceeded

Here is sample of stack trace:

2014-01-02 11:02:49,886 [main] INFO  pl.lrozek.spring.leak.main.LeakMain - 1350 calleeThreadName is SimpleAsyncTaskExecutor-1, callerThreadName is main, areThreadsTheSame: false
[Full GC [PSYoungGen: 39936K->39413K(41984K)] [ParOldGen: 86640K->86551K(87040K)] 126576K->125965K(129024K) [PSPermGen: 57938K->57855K(58368K)], 0.4292090 secs] [Times: user=2.89 sys=0.01, real=0.43 secs] 
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid23319.hprof ...
Heap dump file created [215260624 bytes in 1.822 secs]
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.ClassLoader.getBootstrapResources(ClassLoader.java:1317)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1183)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1181)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(PathMatchingResourcePatternResolver.java:304)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:273)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(PathMatchingResourcePatternResolver.java:339)
	at org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(PathMatchingResourcePatternResolver.java:269)
	at org.springframework.context.support.AbstractApplicationContext.getResources(AbstractApplicationContext.java:1170)
	at org.springframework.context.support.GenericApplicationContext.getResources(GenericApplicationContext.java:223)
	at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:268)
	at org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan(ClassPathBeanDefinitionScanner.java:242)
	at org.springframework.context.annotation.ComponentScanAnnotationParser.parse(ComponentScanAnnotationParser.java:134)
	at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:236)
	at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:205)
	at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:182)
	at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:152)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:299)
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:243)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:254)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:94)
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:609)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464)
	at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
	at pl.lrozek.spring.leak.main.LeakMain.main(LeakMain.java:18)

Here is another stacktrace:

2014-01-02 11:27:58,748 [main] INFO  pl.lrozek.spring.leak.main.LeakMain - 1376 calleeThreadName is SimpleAsyncTaskExecutor-1, callerThreadName is main, areThreadsTheSame: false
[Full GC [PSYoungGen: 39936K->39337K(41984K)] [ParOldGen: 86819K->86819K(87040K)] 126755K->126157K(129024K) [PSPermGen: 55355K->55355K(55808K)], 0.1434080 secs] [Times: user=0.86 sys=0.01, real=0.14 secs] 
[Full GC [PSYoungGen: 39936K->39391K(41984K)] [ParOldGen: 86819K->86819K(87040K)] 126755K->126210K(129024K) [PSPermGen: 55355K->55355K(55808K)], 0.1388310 secs] [Times: user=0.88 sys=0.01, real=0.13 secs] 
[Full GC [PSYoungGen: 39936K->39369K(41984K)] [ParOldGen: 86819K->86763K(87040K)] 126755K->126132K(129024K) [PSPermGen: 55355K->55323K(55808K)], 0.4117840 secs] [Times: user=2.87 sys=0.01, real=0.41 secs] 
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid26941.hprof ...
Heap dump file created [215055088 bytes in 1.795 secs]
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.reflect.Method.copy(Method.java:151)
	at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
	at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
	at java.lang.Class.copyMethods(Class.java:2852)
	at java.lang.Class.getDeclaredMethods(Class.java:1855)
	at org.springframework.util.ReflectionUtils.findMethod(ReflectionUtils.java:154)
	at org.springframework.util.ClassUtils.getMostSpecificMethod(ClassUtils.java:758)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.findResourceMetadata(CommonAnnotationBeanPostProcessor.java:351)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(CommonAnnotationBeanPostProcessor.java:283)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyMergedBeanDefinitionPostProcessors(AbstractAutowireCapableBeanFactory.java:908)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:354)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1094)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:989)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:220)
	at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:618)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
	at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:84)
	at pl.lrozek.spring.leak.main.LeakMain.main(LeakMain.java:18)

Stack traces differ from run to run depending where OOME was thrown. Sometimes exception isn't thrown but than in the console there are only logs of garbage collector activity

java:

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

maven:

Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 17:22:22+0200)
Maven home: /opt/mvn/maven
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: /opt/jvm/jdk1.7.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.0.0-32-generic", arch: "amd64", family: "unix"

Here is the screenshot of the heap when OOME is about to be thrown
!heap leak.png!

---

Affects: 3.2.6, 4.0 GA

Reference URL: https://github.com/lrozek/spring-leak

Attachments:

• heap leak.png (25.26 kB)
• heap leak-mat.png (269.07 kB)

Issue Links:

• Inconsistent hashCode/equals implementation in AbstractPointcutAdvisor results in CGLIB-related memory leak [SPR-12556] #17158 Inconsistent hashCode/equals implementation in AbstractPointcutAdvisor results in CGLIB-related memory leak ("is duplicated by")
• ClassLoader leak with @Async, CGLIB and DefaultListableBeanFactory [SPR-11776] #16398 ClassLoader leak with @Async, CGLIB and DefaultListableBeanFactory
• Method injection causes memory leak [SPR-10785] #15411 Method injection causes memory leak
• @Async with cglib based proxy causes memory leak in permgen [SPR-11276] #15900 @Async with cglib based proxy causes memory leak in permgen

Backported to: 3.2.7

[spring-projects-issues]

Lukasz Rozek commented

The source of the leak is
org.springframework.cglib.proxy.Enhancer -> SOURCE -> cache

cache isn't cleaned after closing application context

!heap leak-mat.png!

[spring-projects-issues]

Juergen Hoeller commented

After some digging, it turns out that the root cause is a missing equals/hashCode implementation in AnnotationMatchingPointcut. That generic pointcut class is rarely used within the framework, since we're using specific pointcut implementations in most cases.

However, Async processing does rely on AnnotationMatchingPointcut. Its lack of a proper 'equals' implementation prevented reuse of any affected proxy class in CGLIB, leading to recreation of identical proxy classes for every new context. Fixed for 4.0.1 and 3.2.7 now.

Juergen