ScalaTest runner issue with SBT, scala.js and Slow test selections

[OndrejSpanel]

I have reported the issue in the SBT repository, but after inspecting the callstack, perhaps it would rather belong here?

sbt/sbt#7184

When using a technique described in How to disable "Slow" tagged Scalatests by default, allow execution with option? to allow running some tests with a dedicated "Slow" configuration, no tests, are run, exception is thrown instead.

The relevant part of the callstack is:

[error] Caused by: java.lang.ClassCastException: class scala.scalajs.js.Object cannot be cast to class scala.scalajs.js.Dictionary (scala.scalajs.js.Object and scala.scalajs.js.Dictionary are in unnamed module of loader sbt.internal.LayeredClassLoader @34a5aab7)
[error]         at scala.scalajs.js.Dictionary$.empty(Dictionary.scala:60)
[error]         at scala.scalajs.reflect.Reflect$.(Reflect.scala:72)
[error]         at org.scalatest.tools.TaskRunner.executionFuture(TaskRunner.scala:81)
[error]         at org.scalatest.tools.TaskRunner.execute(TaskRunner.scala:75)
[error]         at sbt.TestRunner.runTest$1(TestFramework.scala:147)

The error can be reproduced by running sbt Slow/test from a very small repository at https://github.com/OndrejSpanel/ScalaJSSlowTest

[cheeseng]

@OndrejSpanel Sorry for the delay, I am looking into this now to see if I can see anything.

[cheeseng]

@OndrejSpanel @sjrd It seems like the error is coming from Reflect.lookupInstantiatableClass in this line:

https://github.com/scalatest/scalatest/blob/main/js/core/src/main/scala/org/scalatest/tools/TaskRunner.scala#L81

I first thought it was the .asInstanceOf[Suite], but as some investigation it blows up when calling Reflect.lookupInstantiatableClass, and the passed in class name is correct. Also, when the SBT way of selecting the tests is not used, the same class can be loaded and run successfully.

@sjrd I wonder if you may have a better idea what's different in Reflect.lookupInstantiatableClass when using with/without the sbt test selection?

[cheeseng]

@OndrejSpanel @sjrd Fyi I think some side effects happens in inConfig to Reflect.lookupInstantiatableClass, if you remove the Test / testOptions and Slow / testOptions and leave the inConfig line in build.sbt and run sbt test, you can reproduce the same error also.

[cheeseng]

@OndrejSpanel @sjrd It seems to blow up at this 'innocent' line:

https://github.com/scala-js/scala-js/blob/main/library/src/main/scala/scala/scalajs/js/Dictionary.scala#L60

Unfortunately I have no idea what side effect could have caused it.

[sjrd]

The original error message is a JVM error message, not a Scala.js one. Somehow you're trying to run some Scala.js code on the JVM.

[cheeseng]

@sjrd Yes, I think you are right, I am trying to see if it is this line:

https://github.com/OndrejSpanel/ScalaJSSlowTest/blob/master/build.sbt#L12

that's causing the problem and make it to run on JVM instead of JS, if I change it to:

inConfig(Slow)(Defaults.baseTasks),

The error goes away but the filtering does not work also, I wonder if we should use something different in place of Defaults.testTasks for scala-js?

[cheeseng]

@sjrd @OndrejSpanel Using the following inConfig seems to work:

inConfig(Slow)(Defaults.testTasks ++ ScalaJSPlugin.testConfigSettings),

@sjrd Do you mind to verify that it is the correct thing to do?

Thanks.

[sjrd]

Oh, you use a custom Config? Then you also need to add ScalaJSPlugin.testConfigSettings in that inConfig.

Edit: cross-post ^^ Yes, that's the correct thing to do.

[cheeseng]

Thanks @sjrd ! 🙏🏻 @OndrejSpanel can you see if the solution works for you?

[OndrejSpanel]

Yes. it works fine, thanks.

I can see this is documented in https://www.scala-js.org/doc/project/testing.html

Could perhaps something be made to make this more discoverable for people who do not read documentation (I did not), and just hit the error by using the configuration from JVM by blindly copying or using it in a crossproject?