Question: dependency between tests for ParallelTestExecution

[numero-744]

I have tests using an item a of class A. Before the first test, a has to be preprocessed. This preprocessing involves part of the code so it has to be tested (btw it is taking most of the CPU time).

For now I automatically add a preprocessing test before the first test using a. If the preprocessing fails, the prepended test is marked as failed and other tests using a are cancelled.

Example

It is for a DSL.

a should "do something" inSim { dut =>
  // some code
}

a.test("some test") { dut =>
  // some code
}

Creates 3 tests:

• a compiles: preprocessing test
• a should do something: first user test
• a test: some test: second user test

I would like to run tests in parallel, which has two benefits: different instances of A can be preprocessed at the same time, also different tests on a same instance a of class A can be run at the same time.

A solution could be to make sure the prepended test preprocessing a ends before the tests using a start. I cannot find a way to make scalatest enforce such "dependencies" rules.

Another solution could be to group tests, enabling parallelism among test groups but not among tests of the same test group. It would not make it possible to run tests using the same A at the same time but it is not an issue because preprocessing a is the critical thing. I also have no clue about how to do that.

[cheeseng]

@numero-744 Sorry for responding late, I wonder if there's a way to do this via nested suites in combination of StepwiseNestedSuiteExecution, here's some info about nested suites:

https://www.scalatest.org/scaladoc/3.2.14/org/scalatest/Suite.html

In the overriden nestedSuites function perhaps you can pass a value produced by ASuite, to instantiate BSuite, and in BSuite run or cancel the tests based on the passed in value. If ASuite is a async suite, you may also pass the Status of run from ASuite into BSuite, and ask it to wait until ASuite is completed before getting the value computed from ASuite.

Anyway, just some idea via current nested suites feature. I remember some discussion with @bvenners on similar requirement before, but I can't recall we implemented anything for it yet.

Also, if a is ok to be mutable ScalaTest's 'path' style trait may help:

https://www.scalatest.org/scaladoc/3.2.14/org/scalatest/freespec/PathAnyFreeSpec.html

[numero-744]

No worry,

So I could have two suites Compilations and Simulations, and ensure that all tests in Compilations are run before tests in Simulations start. However I have no idea about how to dynamically build these two test suites.

My current implementation is internally using AnyFreeSpec (extending it but not requiring the user to know that) to nest things using "text" - { code }, which uses context and I would like to continue to use it as it is handy. I could collect tests in lists and iterate over these lists but guess I would loose information about nesting automatically collected by Scalatest?

Also I can fall on several components with the same name (in different contexts) so if I gather them all at the beginning without the original nesting context, I'll have the "several tests with the same name" error…

---

More context about the current use-case can help (but I don't want to take too much of your time so don't read it if you think it is not relevant):

This work is for a hardware description DSL. In hardware we work on a device we want to test (named "device under test", DUT). These devices can have variants (different configurations) or several implementations to do the same thing. I provide a way for the user to define different groups of tests (some test suites) so that the user can run a test suite with a given device (with a specific configuration).

So I also have some nesting. Here is a complete empty example using this nesting:

class EmptyTestSuite extends SpinalTestBench {
  // define some components with different configurations
  val identity8 = Dut(Identity(IdentityConfig(8)))
  val identity10 = Dut(Identity(IdentityConfig(10)))

  // define an "abstract" test suite to test my components
  val testSuite1 = TestSuite[Identity] { it =>
    it should "do something good" inSim { dut => /* here we don't care about content of the tests */ }
    it should "do something else" inSim { dut => /* here we don't care about content of the tests */ }
  }

  // run the test suite with my components
  testSuite1.runWith(identity8, identity10)
}

I could have several test suites, and I get this nesting:

EmptyTestSuite:
testSuite1
  identity8
  - identity8 compiles
  - identity8 should do something good
  - identity8 should do something else
  identity10
  - identity10 compiles
  - identity10 should do something good
  - identity10 should do something else

To make things go parallel, the dependencies would be:

• identity8 compiles -> identity8 should do something good
• identity8 compiles -> identity8 should do something else
• identity10 compiles -> identity10 should do something good
• identity10 compiles -> identity10 should do something else