test.sequential for tests that rely on the sequence

[turadg]

The problem

In our organization the vast majority of uses of test.serial are just to enforce that the tests don't run in parallel. They can still be run independently with the -m to select.

In some of our tests, test.serial is used for tests that depend on a sequence of execution. For example, a scenario test in which there are phases (e.g. A, B, C). Each test is both testing a phase and setting up the next phase. We could turn it into one giant test, but they are logically independent and a failure in A can allow B to still be tested and pass. We could have multiple tests like A[setupA,testA] and B[setupA,setupB,testB] but these setups are very slow and we want to do each only once.

Proposal

Maintain test.serial as "not parallel". Consider randomizing the order so that it's not relied upon to be executed in sequence.

Add a test.sequential for tests that actually do depend on the sequence. Selecting one of these with "-m" executes the preceding tests.

[novemberborn]

Add a test.sequential for tests that actually do depend on the sequence.

My first thought is that would make randomization harder to implement.

Selecting one of these with "-m" executes the preceding tests.

And that is a lot harder to make happen.

I'm not sure it's worthwhile.

these setups are very slow and we want to do each only once.

Could you elaborate on the setup logic? Maybe you could use shared workers to only have to do the setup once?

[turadg]

The setup is slow and stateful. It takes about thirty seconds to spin up what's basically a virtual machine with an OS running. Test A is something like "install this program, run it, and verify it works correctly". Several tests are just this. Then we have more use cases to test and they're predicated on the state after A so a B is added that says "install this other program that depends on the first, run it, and verify it works correctly".

If there were some way to fork a test state that would be ideal. We could setupA, then fork it for testA and setupB.

Given the difficulty of implementing what I proposed, here's a simpler option:
a way to specify within a test file whether its test.serial to follow are to be randomized. Then the scenario tests would opt for sequential (the status quo) and the independent tests would opt for randomized.

[novemberborn]

AVA's shared workers are great for this. You can spin up the VM in AVA's main process and then communicate details such as ports to the test workers. https://github.com/avajs/ava/blob/main/docs/recipes/shared-workers.md

[turadg]

I think you're proposing shared workers because I said "basically a virtual machine". To be clear, it's an entirely JS one so I don't need ports. My point was it has a lot of in-memory state and that is slow to reach.)

I had considered shared works but maybe I missed something: how could they save repeating setupA for testA and testB?

[novemberborn]

I brought it up because shared workers are only torn down when AVA exits, so they can hold ok to your setup and make it available to tests.

Whether that applies to your use case I don't know. You're restricted to message passing so it's great for servers or build steps but not if you need to invoke a function.