Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous deserialization performed by Hazelcast may fail due to the wrong ClassLoader being used #24836

Closed
dsyer opened this issue Jan 14, 2021 · 12 comments
Assignees
Labels
type: bug A general bug
Milestone

Comments

@dsyer
Copy link
Member

dsyer commented Jan 14, 2021

I discovered this in a long yak-shaving session migrating an app to Boot 2.4.1 and Hazelcast 4.0.3.

This fails when you run it from the jar:

@SpringBootApplication
public class IssueApplication {
	public static void main(String[] args) {
		SpringApplication.run(IssueApplication.class, args);
	}
	@Bean
	CommandLineRunner runner(HazelcastInstance hazelcast) {
		IMap<String, Foo> sessions = hazelcast.getMap("foos");
		return args -> {
			sessions.set("foo", new Foo("foo"));
			System.err.println(sessions.get("foo"));
			sessions.getAsync("foo").whenComplete((u, t) -> System.err.println(u)).toCompletableFuture().get();
		};
	}
}
class Foo implements Serializable {
	private String value;
	public Foo() {
	}
	public Foo(String value) {
		this.value = value;
	}
	public String getValue() {
		return this.value;
	}
	public void setValue(String value) {
		this.value = value;
	}
	@Override
	public String toString() {
		return "Foo [value=" + this.value + "]";
	}
}

It's fine when you run in an IDE, or with java -classpath .... The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.

I worked around it with this

		@Bean
		Config hazelcastConfig() {
			Config config = new Config();
			config.setClassLoader(SpringApplication.class.getClassLoader());
			...
			return config;
		}

but the issue is more generic really - probably any library that uses JDK fork-join utilities will end up with the same class loader.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Jan 14, 2021
@wilkinsona
Copy link
Member

The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.

I assume that JarLauncher here should be LaunchedURLClassLoader. I think this may also be the wrong way round. In a Boot app you want the TCCL to be the LaunchedURLClassLoader as this is the class loader the can load the application's classes and those of its dependencies. The AppClassLoader can only see the launcher and the JDK.

Unfortunately, the JDK's common fork-join pool uses the app class loader as its TCCL. This makes it unsuitable for use in a packaged Boot application and, I suspect, in any environment with a custom class loader such as a Servlet container. We've seen this problem before (#15737) and, in general terms, I don't think we can do anything about it.

In this specific case, it looks like the auto-configuration could be improved by setting the class loader automatically (assuming that's possible while still honouring the rest of the user's configuration).

@dsyer
Copy link
Member Author

dsyer commented Jan 14, 2021

Hmm. I think maybe there is still something in Hazelcast which we could ask them to fix.

This app works fine (IDE and JAR) when Hazelcast is not on the classpath:

@SpringBootApplication
public class IssueApplication {

	public static void main(String[] args) {
		SpringApplication.run(IssueApplication.class, args);
	}

	@Bean
	CommandLineRunner runner() {
		return args -> {
			ForkJoinPool.commonPool().submit(() -> {
				System.err.println(ClassUtils.resolveClassName(Foo.class.getName(), null));
			}).get();
		};
	}

}
...

and then it fails from a JAR if you just put Hazelcast on the classpath but never use it:

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-01-14 14:46:30.747 ERROR 1000765 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:788) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1311) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	at com.example.IssueApplication.main(IssueApplication.java:16) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006) ~[na:na]
	at com.example.IssueApplication.lambda$runner$1(IssueApplication.java:24) ~[classes!/:0.0.1-SNAPSHOT]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
	... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[na:na]
	... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
	at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:334) ~[spring-core-5.3.3.jar!/:5.3.3]
	at com.example.IssueApplication.lambda$runner$0(IssueApplication.java:23) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[na:na]
Caused by: java.lang.ClassNotFoundException: com.example.Foo
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
	at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
	at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.3.jar!/:5.3.3]
	at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.3.3.jar!/:5.3.3]
	... 7 common frames omitted

Adding Hazelcast to the classpath (and an empty hazelcast.xml) causes our autoconfiguration to be created, which in turn seems to mess with the default ForkJoinPool.

@wilkinsona
Copy link
Member

That's odd. The common pool uses DefaultForkJoinWorkerThreadFactory which "creates a new ForkJoinWorkerThread using the system class loader as the thread context class loader". I don't know how Hazelcast could be messing with that. Sounds like some digging is required.

@dsyer
Copy link
Member Author

dsyer commented Jan 14, 2021

This is even weirder. If you have a 2 node cluster it doesn't fail from the JAR. It only fails with one node.

@dsyer
Copy link
Member Author

dsyer commented Jan 14, 2021

That's rubbish, sorry. Brain fart probably. It always fails from the JAR for the reason you said (the system class loader is the wrong one).

UPDATE: it sometimes fails. Especially with Java 8 (as opposed to 11, where ForkJoinPool always uses the system class loader).

@dsyer
Copy link
Member Author

dsyer commented Jan 14, 2021

So maybe we need to set the class loader in Hazelcast Config in our autoconfig?

@wilkinsona wilkinsona changed the title Thread context class loader ends up being wrong when launched from jar Asynchronous deserialization performed by Hazelcast may fail due to the wrong ClassLoader being used Jan 14, 2021
@wilkinsona wilkinsona added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged labels Jan 14, 2021
@wilkinsona wilkinsona added this to the 2.4.x milestone Jan 14, 2021
@wilkinsona
Copy link
Member

Yep, I think so. Now we need to figure out how to do that across all the various configuration options that we support.

@dsyer
Copy link
Member Author

dsyer commented Jan 14, 2021

I think they all end up in a Config object somehow. There's a utility in HC, or something, for loading the XML. And the Config is mutable, so we can just create it in whatever way we find the user is trying to suggest and set the class loader. Maybe?

@HJK181

This comment was marked as off-topic.

@wilkinsona

This comment was marked as off-topic.

@dsyer

This comment was marked as off-topic.

@HJK181

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

No branches or pull requests

5 participants