ava.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess #8916

jz499 · 2019-03-05T03:11:51Z

I am trying to start a Spark Session but during the initialization process I'm consistently getting errors for netty, version 4.0.56 also exhibited the same behavior.

Expected behavior

No Errors

Actual behavior

java.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess at io.netty.util.internal.shaded.org.jctools.queues.LinkedArrayQueueUtil.modifiedCalcElementOffset(LinkedArrayQueueUtil.java:24) at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.poll(BaseMpscLinkedArrayQueue.java:326)
at io.netty.util.concurrent.SingleThreadEventExecutor.pollTaskFrom(SingleThreadEventExecutor.java:208)
at io.netty.util.concurrent.SingleThreadEventExecutor.pollTask(SingleThreadEventExecutor.java:203)
at io.netty.channel.nio.NioEventLoop.pollTask(NioEventLoop.java:523)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:393)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:859)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:955)
19/03/04 21:38:37 WARN channel.AbstractChannel: Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x4d268a31]
java.lang.ExceptionInInitializerError
at io.netty.util.internal.shaded.org.jctools.queues.LinkedArrayQueueUtil.modifiedCalcElementOffset(LinkedArrayQueueUtil.java:24)
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.offer(BaseMpscLinkedArrayQueue.java:308)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:329)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:479)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331)
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282)
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278)
at org.apache.spark.network.server.TransportServer.init(TransportServer.java:131)
at org.apache.spark.network.server.TransportServer.(TransportServer.java:73)
at org.apache.spark.network.TransportContext.createServer(TransportContext.java:114)
at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:115)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:466)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:464)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2275)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:161)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2310)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:469)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
at org.apache.spark.SparkContext.(SparkContext.scala:423)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
at cli.javo_net.myclass.mymethod(Program.cs:52)
at cli.javo_net.Program.Main(Program.cs:37)
Caused by: java.lang.IllegalStateException: Unknown pointer size
at io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess.(UnsafeRefArrayAccess.java:51)
... 29 more
19/03/04 21:38:37 ERROR DefaultPromise.rejectedExecution: Failed to submit a listener notification task. Event loop shut down?
java.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess
at io.netty.util.internal.shaded.org.jctools.queues.LinkedArrayQueueUtil.modifiedCalcElementOffset(LinkedArrayQueueUtil.java:24)
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.offer(BaseMpscLinkedArrayQueue.java:308)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:329)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:434)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:491)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331)
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282)
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278)
at org.apache.spark.network.server.TransportServer.init(TransportServer.java:131)
at org.apache.spark.network.server.TransportServer.(TransportServer.java:73)
at org.apache.spark.network.TransportContext.createServer(TransportContext.java:114)
at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:115)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:466)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:464)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2275)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:161)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2310)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:469)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
at org.apache.spark.SparkContext.(SparkContext.scala:423)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
at cli.javo_net.myclass.mymethod(Program.cs:52)
at cli.javo_net.Program.Main(Program.cs:37)
19/03/04 21:38:37 ERROR DefaultPromise.rejectedExecution: Failed to submit a listener notification task. Event loop shut down?
java.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess
at io.netty.util.internal.shaded.org.jctools.queues.LinkedArrayQueueUtil.modifiedCalcElementOffset(LinkedArrayQueueUtil.java:24)
at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.offer(BaseMpscLinkedArrayQueue.java:308)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:329)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:434)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:123)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:494)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331)
at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282)
at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278)
at org.apache.spark.network.server.TransportServer.init(TransportServer.java:131)
at org.apache.spark.network.server.TransportServer.(TransportServer.java:73)
at org.apache.spark.network.TransportContext.createServer(TransportContext.java:114)
at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:115)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:466)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:464)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2275)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:161)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2310)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:469)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
at org.apache.spark.SparkContext.(SparkContext.scala:423)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
at cli.javo_net.myclass.mymethod(Program.cs:52)
at cli.javo_net.Program.Main(Program.cs:37)

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

netty-all-4.1.17.Final.jar

JVM version (e.g. `java -version`)

java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)

OS version (e.g. `uname -a`)

Windows10

The text was updated successfully, but these errors were encountered:

normanmaurer · 2019-03-05T08:02:21Z

Please upgrade to the latest 4.1.x version and report back if this still happens.

jz499 · 2019-03-05T15:48:14Z

I have upgraded to the latest build "netty-all-4.1.34.Final-20190305.083539-41", and I still get the same errors.

normanmaurer · 2019-03-05T18:11:32Z

this is strange... Seems like it is caused by jctools.

@nitsanw any idea:

Caused by: java.lang.IllegalStateException: Unknown pointer size
at io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess.(UnsafeRefArrayAccess.java:51)

nitsanw · 2019-03-06T12:56:40Z

@jz499 This is very strange... Java refs are either 4b (compressed oops) or 8b, there's no other option AFAIK. Can you check if jctools can build cleanly on that machine?

jz499 · 2019-03-06T17:24:15Z

I have tried several version of JCTools and netty but the problem still persists, I did find another reference online for this issue. https://sourceforge.net/p/ikvm/mailman/message/35520859/
Please review the link it seems to be a valid root cause analysis.

nitsanw · 2019-03-06T17:48:35Z

The link is suggesting missing intrinsics, but these are established on 8u192 AFAIK. Can you build JCTools on the machine in question?

jz499 · 2019-03-06T19:38:37Z

Yes, I'm able to buid JCTools with 1 minor error.

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for JCTools Parent 2.2-SNAPSHOT:
[INFO]
[INFO] JCTools Parent ..................................... SUCCESS [ 11.163 s]
[INFO] JCTools Build Tools ................................ SUCCESS [ 8.035 s]
[INFO] Java Concurrency Tools Core Library ................ SUCCESS [04:45 min]
[INFO] Channel implementations ............................ FAILURE [ 3.298 s]
[INFO] Experimental implementations ....................... SKIPPED
[INFO] Benchmarks ......................................... SKIPPED
[INFO] concurrency-test ................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:20 min
[INFO] Finished at: 2019-03-06T14:22:12-05:00

normanmaurer · 2019-03-15T09:15:17Z

@nitsanw ^^ any idea ?

nitsanw · 2019-03-15T12:29:48Z

Sorry, I'm baffled as to why @jz499 would be able to run the build on the same machine and not hit this issue. The master JCTools now reports the ref size:
https://github.com/JCTools/JCTools/blob/81f135fa32ea53f1511114bb6dfdc87d1730b8a1/jctools-core/src/main/java/org/jctools/util/UnsafeRefArrayAccess.java#L51

Perhaps try with that version.

jz499 · 2019-03-15T18:55:59Z

Is there a netty build with this new/modified jctool?

jz499 · 2019-03-16T03:37:53Z

It seems to be a problem with IKVM rather than netty (see link below)

https://stackoverflow.com/questions/50518771/netty-exceptionininitializererror

normanmaurer · 2019-04-01T06:49:35Z

@jz499 btw as workaround you may want to use -Dio.netty.noUnsafe=true and see if this works for you

normanmaurer · 2019-04-12T18:33:30Z

Should be fixed by #9042 for now

Motivation: IKVM.NET seems to ship a bug sun.misc.Unsafe class, for this reason we should better disable our sun.misc.Unsafe usage when we detect IKVM.NET is used. Modifications: Check if IKVM.NET is used and if so do not use sun.misc.Unsafe by default. Result: Fixes #9035 and #8916.

jz499 · 2019-04-29T14:25:59Z

Versions 4.1.35.Final and 4.1.36.Final resolved the netty errors I had back in March. I'm now able to build my project successfully. Thank You.

Motivation: IKVM.NET seems to ship a bug sun.misc.Unsafe class, for this reason we should better disable our sun.misc.Unsafe usage when we detect IKVM.NET is used. Modifications: Check if IKVM.NET is used and if so do not use sun.misc.Unsafe by default. Result: Fixes (netty#9035) and (netty#8916).

johnou added a commit to johnou/JCTools that referenced this issue Mar 6, 2019

Add scale to exception message to help debug netty/netty#8916

81f135f

johnou mentioned this issue Mar 6, 2019

Add scale to exception message to help debug JCTools/JCTools#237

Merged

normanmaurer mentioned this issue Apr 12, 2019

Don't use sun.misc.Unsafe when IKVM.NET is used #9042

Merged

normanmaurer added this to the 4.1.35.Final milestone Apr 12, 2019

normanmaurer modified the milestones: 4.1.35.Final, 4.1.36.Final Apr 17, 2019

normanmaurer modified the milestones: 4.1.36.Final, 4.1.35.Final Apr 29, 2019

normanmaurer closed this as completed Apr 29, 2019

nitsanw mentioned this issue Oct 13, 2022

Netty + IKVM.NET = crash #12847

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ava.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess #8916

ava.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess #8916

jz499 commented Mar 5, 2019

normanmaurer commented Mar 5, 2019

jz499 commented Mar 5, 2019

normanmaurer commented Mar 5, 2019

nitsanw commented Mar 6, 2019

jz499 commented Mar 6, 2019

nitsanw commented Mar 6, 2019

jz499 commented Mar 6, 2019

normanmaurer commented Mar 15, 2019

nitsanw commented Mar 15, 2019

jz499 commented Mar 15, 2019

jz499 commented Mar 16, 2019

normanmaurer commented Apr 1, 2019

normanmaurer commented Apr 12, 2019

jz499 commented Apr 29, 2019

ava.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess #8916

ava.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess #8916

Comments

jz499 commented Mar 5, 2019

Expected behavior

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

JVM version (e.g. java -version)

OS version (e.g. uname -a)

normanmaurer commented Mar 5, 2019

jz499 commented Mar 5, 2019

normanmaurer commented Mar 5, 2019

nitsanw commented Mar 6, 2019

jz499 commented Mar 6, 2019

nitsanw commented Mar 6, 2019

jz499 commented Mar 6, 2019

normanmaurer commented Mar 15, 2019

nitsanw commented Mar 15, 2019

jz499 commented Mar 15, 2019

jz499 commented Mar 16, 2019

normanmaurer commented Apr 1, 2019

normanmaurer commented Apr 12, 2019

jz499 commented Apr 29, 2019

JVM version (e.g. `java -version`)

OS version (e.g. `uname -a`)