Possibilities to estimate the total test duration before starting the first test?

[maettu-this]

Thought somebody else already tried this, but found nothing.

I have a bunch of tests communicating with speed limited embedded systems, resulting in total test duration of several hours. In order to better handle this, I have marked the tests with a DurationCategoryAttribute that states the approximate duration of all long tests. For even better handle this, I have tried to reflect on these attributes to sum up an estimate of the total test duration, considering input from #3627.

However, as my project spans across multiple assemblies, and NUnit loads one assembly after each other, it seems impossible to estimate the duration across all test assemblies before the first test gets started. #4106 doesn't seem to offer any alternative approach to get this information either.

I am aware of the limitations of estimating the total test duration, e.g. Parallelizable vs. sequential test execution. However, the available connected devices limit the level of parallelism. And filtering the tests for a given device will for sure result in sequential test execution, which would enable calculating the estimate for each set, rather than just running it each night and then see how long it took.

Is there anything I have missed, an NUnit feature, an issue discussion or piece of documentation that could be used to achieve such estimation?

[OsirisTerje]

Can you elaborate on what is your specific ask here?

Do you want to see all attributes from a test class/method in the output stream? That is including your own custom attribute?
Do you want to see whether classes are for parallel execution or not?

Beside that, NUnit doesn't have any mechanism for such estimation OOB.

[maettu-this]

Potential approach:

1. NUnit can be configured to load all test assemblies within a project before starting executing anything.

Upon starting a test run:

2. NUnit provides a "global" (i.e. "process-wide" and "cross-assembly") SetUpFixture that is called after having loaded all test assemblies.
3. The "global" SetUpFixture can retrieve information about all loaded test assemblies.
4. The information includes the list of all test fixtures/test methods that are about to be executed.
5. The global SetUpFixture can then reflect on the attributes (e.g. my DurationCategoryAttribute) of all test fixtures or test methods enabled for execution.
6. The global SetUpFixture can estimate the expected test duration and potentially act upon the result, e.g. issue a warning if a test run takes longer than a specified maximum amount of time or even cancel the test run.

Maybe such approach is far from today's possibilities. Maybe most or even all is already there. Any input and thoughts are appreciated.

[OsirisTerje]

Perhaps you can use the NUnitLite runner (because it is simple and direct) and just use the Explore command. That way you would get that loaded list of test assemblies and the tests they contain. You can modify the code there to dump the list you require.

The only way to add a cross-assembly setupfixture that would work, would be to use either a custom runner, or NUnitLite. You could possibly use the NUnit.Console, but that adds another layer.

I believe that modifying a runner would be easier than trying the Setupfixture road.

[maettu-this]

I see, I may give that a try. Does the NUnitLite runner support categories? https://docs.nunit.org/articles/nunit/running-tests/NUnitLite-Options.html doesn't mention such option anymore, while the legacy wiki at https://github.com/nunit/docs/wiki/NUnitLite-Options/ec67ba644536af8c82c0c537d2f2d84bc08312b3 mentions --include=CATEGORIES and --exclude=CATEGORIES.

[OsirisTerje]

Yes, it supports categories. Look at the Where clause.

[maettu-this]

Based on the output of an experimental run with --explore as well as #344 it looks like NUnitLite is limited to a single assembly. The comment...

see it's purpose as enabling self-executing tests for a single test assembly

...in #2124 supports the finding. However, @OsirisTerje you stated...

get that loaded list of test assemblies and the tests they contain

...which indicates this might have be available somehow. But I haven't figured out how. Any more thoughts are highly appreciated.

[maettu-this]

Digging a bit further, #3202 contains some additional input but also indicates that neither the NUnit framework nor any of the runners (incl. NUnitLite) does support iterating over multiple assemblies before starting the first test.

Further looking at the NUnitLite options I guess it could be an approach to add a --testfiles=PATH option that specifies assemblies to include in a run. This would allow retrieving information about multiple test assemblies before starting a run.

I conclude there currently is no feasible way to achieve what I would like. Feel free on how to move ahead with this issue:

• Close without action
• Keep open as is
• Rename to e.g. „obtain a list of all test assemblies, fixtures and tests before starting a test run"

[OsirisTerje]

I assume you are aware that nunitlite makes the test assembly into a self executing test. At start, it calls into the nunitlite runner. So it us the test assembly that is calling the runner. It then doesn't make sense to call other assemblies, besides letting one test assembly be a parent and adding references to the others. The other option is to create a small shell script to call the different nunitlite enabled test assemblies and executing them.

[maettu-this]

Exactly, as initially stated "my project spans across multiple assemblies" there indeed is a "main" test assembly which is at the top of the chain, similar as the application's "main" assembly directly or indirectly references all underlying assemblies. That's the reason behind the 3rd proposal to rename this issue to "obtain a list of all test assemblies, fixtures and tests before starting a test run". Whether the approach is based on the NUnitLite runner or a "global" (i.e. "process-wide" and "cross-assembly") SetUpFixture or something else doesn't matter to my use case. But as stated this morning, it's fully OK if NUnit decides to not intend to support this.

[OsirisTerje]

If your structure is like that, you don't need any list if assemblies. It will execute all your tests, since you reference them.

[maettu-this]

That's interesting, then either me doing something wrongly, or NUnitLite or NUnit doesn't behave as expected. Here's a repro project:

Issue4696.zip

Each of the 3 projects contains 1 test. The TestMain references the other 2 projects. As visible in the below result, running \TestMain\bin\Debug\TestMain.exe only identifies 1 of the 3 tests:

PS D:\Workspace\NUnit\Issue4696\TestMain\bin\Debug> .\TestMain.exe
NUnitLite 3.14.0 (.NET Framework 4.5)
Copyright (c) 2023 Charlie Poole, Rob Prouse

Runtime Environment
   OS Version: Microsoft Windows NT 10.0.19045
  CLR Version: 4.0.30319.42000

Test Files
    D:/Workspace/NUnit/Issue4696/TestMain/bin/Debug/TestMain.exe

Test Discovery
  Start time: 2024-05-19 18:42:00Z
    End time: 2024-05-19 18:42:00Z
    Duration: 0.030 seconds

Run Settings
    Number of Test Workers: 16
    Work Directory: D:\Workspace\NUnit\Issue4696\TestMain\bin\Debug
    Internal Trace: Off

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2024-05-19 18:42:00Z
    End time: 2024-05-19 18:42:00Z
    Duration: 0.039 seconds

Results (nunit3) saved as D:\Workspace\NUnit\Issue4696\TestMain\bin\Debug\TestResult.xml

[OsirisTerje]

Hi! Sorry about not being clear here.

I moved your repro into the nunit.issues repo, here: https://github.com/nunit/nunit.issues/tree/main/Issue4696

The point is that you need to address the test fixtures. That is all that is required.
I have done that in the repro code, and the results comes out as:

The way it is done is to just inherit the test classes (aka test fixtures). This inherited classes don't need to do anything, so they could also be auto-generated. They will provide the necessary binding, and sort of "move" the tests into the correct scope.

Code added is then:

 public class T1 : Issue4696.TestProject1.Tests
 {}

 public class T2 : Issue4696.TestProject2.Tests
 {}

[maettu-this]

I see, thanks for clarifying. Unfortunately this isn't really practical with ~150 test fixtures in ~15 test assemblies. Still, the pieces of information provided could help others having just a few assemblies and fixtures to achieve something similar.