Application does not start with BlockHound and ReactorDebugAgent installed on java 8 #70

robotmrv · 2019-11-03T16:31:30Z

BlockHound and ReactorDebugAgent cannot work together on java 8.
Application fails with error

java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.WindowsAttachProvider could not be instantiated
java.lang.ExceptionInInitializerError
Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@53aac487
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:27)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers installed
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:203)
	... 11 more
Exception in thread "main"

If I try to start application using java 11 or just disable BlockHound or ReactorDebugAgent (in the code and spring-boot integration) everything works fine.

Expected Behavior

application starts

Actual Behavior

application start fails

Steps to Reproduce

try to start application on java 8
https://github.com/robotmrv/reactor-debug-error

Possible Solution

Your Environment

Reactor version(s) used:
io.projectreactor:reactor-core:3.3.0.RELEASE
io.projectreactor:reactor-tools:3.3.0.RELEASE
io.projectreactor.tools:blockhound:1.0.1.RELEASE
Other relevant libraries versions (eg. netty, ...):
JVM version (javar -version):
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
OS and version (eg uname -a):
Win 10

The text was updated successfully, but these errors were encountered:

bsideup · 2019-11-03T16:36:47Z

HI @robotmrv,

Thanks for reporting!
Have you tried different combinations, e.g. "only BlockHound" or "only reactor-tools?

Just trying to understand whether it fails only if both are installed.

robotmrv · 2019-11-03T16:48:19Z

Hi @bsideup,
If only one of BlockHound / ReactorDebugAgent is installed it works as expected

bsideup · 2019-11-05T13:11:00Z

@robotmrv does it happen with BlockHound 1.0.0.RELEASE?

robotmrv · 2019-11-05T13:43:49Z

@bsideup yes, the same with BlockHound 1.0.0.RELEASE

bsideup · 2019-11-05T14:38:34Z

@robotmrv thanks for verifying! 👍 on it...

robotmrv · 2019-11-08T09:56:33Z

Hi @bsideup
I've tested example application using linux based docker image openjdk:8

Linux c199ba7a5c0d 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 GNU/Linux
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

and got similar result.

So it is not Windows specific

 09:37:42.177 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
 Exception in thread "main" java.lang.ExceptionInInitializerError
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
      at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
      at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
 Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:328)
      at reactor.blockhound.BlockHound.install(BlockHound.java:94)
      at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
      ... 8 more
 Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@311d617d
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:608)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:581)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:533)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:510)
      at reactor.blockhound.BlockHound$Builder.install(BlockHound.java:309)
      ... 10 more
 Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at reactor.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:106)
      at reactor.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:603)
      ... 14 more
 Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1900)
      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1838)
      at java.lang.Runtime.loadLibrary0(Runtime.java:870)
      at java.lang.System.loadLibrary(System.java:1122)
      at sun.tools.attach.LinuxVirtualMachine.<clinit>(LinuxVirtualMachine.java:342)
      at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
      at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
      ... 20 more

bsideup · 2019-11-08T13:27:00Z

Caused by: java.lang.UnsatisfiedLinkError: Native Library /usr/local/openjdk-8/jre/lib/amd64/libattach.so already loaded in another classloader

Interesting...

/cc @raphw

raphw · 2019-11-08T14:09:15Z

Probably a bieffect of shading with the way attach works on Java 8 where tools.jar must be loaded into a new class loader. Byte Buddy checks if it already loaded tools.jar previously and reuses that attachment if possible but with the shading, I assume another, unshaded copy of Byte Buddy attempts the same what is impossible in Java 8 as the native code in tools.jar can only loaded once. What other tools you use load tools.jar? (Set a break point in com.sun.tools.attach.VirtualMachine.)

The solutions:

Use emulated attach (add JNA). It does not set any VM global state.
Avoid shading Byte Buddy if an unshaded copy is attempting the attach.
Upgrade to Java 9+ where this is fixed in the JVM.

bsideup · 2019-11-11T10:52:54Z

@raphw thank you for the explanation!

I just tried adding JNA but it still fails, this time with:

Caused by: java.lang.IllegalStateException: Error during attachment using: reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@66498326
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:400)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
	at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41)
	at com.example.demo.ReactorDebugApplication.<clinit>(ReactorDebugApplication.java:22)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at reactor.tools.shaded.net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:395)
	... 5 more
Caused by: java.lang.UnsatisfiedLinkError: Native Library /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/libattach.dylib already loaded in another classloader
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1845)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at sun.tools.attach.BsdVirtualMachine.<clinit>(BsdVirtualMachine.java:307)
	at sun.tools.attach.BsdAttachProvider.attachVirtualMachine(BsdAttachProvider.java:63)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	... 11 more

raphw · 2019-11-12T02:22:03Z

In this case you need to tell Byte Buddy explicitly to use attachment emulation rather then the built-in attachment. It will always prefer the latter if available and this exception occurs a bit too late in the process to fall back to something else, unfortunately.

bsideup · 2019-11-20T14:18:22Z

It seems that there is no easy fix we can do in BlockHound or reactor-tools and we're affected by raphw/byte-buddy#670, the same way as seen in Kotlin/kotlinx.coroutines#1060

As a workaround, I would suggest using Java 9+ for running the tests

bsideup · 2019-11-20T14:23:40Z

Also Mockk:
mockk/mockk#15

robotmrv · 2020-09-20T20:57:02Z

@bsideup is it possible to add original artifact without shading (the same way as reactor-tools publishes it now). It seems that usage of the same non-shaded ByteBuddyAgent should solve the problem

bsideup · 2020-10-19T15:40:54Z

@robotmrv yeah, I think we could do that. Alternatively, JNA can be added to classpath to make BB emulate the attachment.

bsideup changed the title ~~Application does not start with BlockHound and ReactorDebugAgent installed on java 8~~ Application does not start on Windows with BlockHound and ReactorDebugAgent installed on java 8 Nov 3, 2019

bsideup added area/java-agent status/need-investigation This needs more in-depth investigation type/bug A general bug labels Nov 3, 2019

bsideup added this to the next milestone Nov 3, 2019

bsideup changed the title ~~Application does not start on Windows with BlockHound and ReactorDebugAgent installed on java 8~~ Application does not start with BlockHound and ReactorDebugAgent installed on java 8 Nov 8, 2019

bsideup added status/on-hold This was set aside or cannot be worked on yet status/has-workaround This has a known workaround described and removed status/need-investigation This needs more in-depth investigation labels Nov 20, 2019

bsideup removed this from the next milestone Jan 23, 2020

bsideup mentioned this issue Oct 19, 2020

Mockito.mock fails with BlockHound #152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Application does not start with BlockHound and ReactorDebugAgent installed on java 8 #70

Application does not start with BlockHound and ReactorDebugAgent installed on java 8 #70

robotmrv commented Nov 3, 2019

bsideup commented Nov 3, 2019

robotmrv commented Nov 3, 2019

bsideup commented Nov 5, 2019

robotmrv commented Nov 5, 2019

bsideup commented Nov 5, 2019

robotmrv commented Nov 8, 2019

bsideup commented Nov 8, 2019

raphw commented Nov 8, 2019 •

edited

bsideup commented Nov 11, 2019

raphw commented Nov 12, 2019

bsideup commented Nov 20, 2019

bsideup commented Nov 20, 2019

robotmrv commented Sep 20, 2020

bsideup commented Oct 19, 2020

Application does not start with BlockHound and ReactorDebugAgent installed on java 8 #70

Application does not start with BlockHound and ReactorDebugAgent installed on java 8 #70

Comments

robotmrv commented Nov 3, 2019

Expected Behavior

Actual Behavior

Steps to Reproduce

Possible Solution

Your Environment

bsideup commented Nov 3, 2019

robotmrv commented Nov 3, 2019

bsideup commented Nov 5, 2019

robotmrv commented Nov 5, 2019

bsideup commented Nov 5, 2019

robotmrv commented Nov 8, 2019

bsideup commented Nov 8, 2019

raphw commented Nov 8, 2019 • edited

bsideup commented Nov 11, 2019

raphw commented Nov 12, 2019

bsideup commented Nov 20, 2019

bsideup commented Nov 20, 2019

robotmrv commented Sep 20, 2020

bsideup commented Oct 19, 2020

raphw commented Nov 8, 2019 •

edited