ensure order of tests whenever using a dataProvider

[bitcoder]

Hi everyone,
I want to ensure that tests generated using a DataProvider, as shown ahead, are executed by order.
The execution order seems that is not guarantee (sometimes it executes in one way, sometimes in another way); in the docs I wasn't able to find an explicit statement but I did find this issue which may (?) be related.
Anway, I want to ensure the items returns by the DataProvider to be executed by order. How can I ensure that? Do I need to refactor this code somehow?
Thanks in adbaance.

public class DataDrivenExamples {
 
    @DataProvider(name = "numbers")
    public static Object[][] evenNumbers() {
        // the last data/example if not correct on purpose, to fail the test
        return new Object[][]{{1, false}, {2, true}, {4, true}, {5, true}};
    }
  

    @Test(dataProvider = "numbers")
    public void givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect(Integer number, boolean expected) {
        assertEquals(expected, number % 2 == 0);
    }

}

Note: tested using TestNG 7.9.0

[itkhanz]

@krmahadevan could you please have a look at this discussion and sample code-snippet and suggest if there's a way to preserve order of tests in this scenario?

[itkhanz]

Thank you @krmahadevan for details response. We solved our issue by sorting the results.

Below you will find the details of ticket which was causing flakiness of the build due to unexpected ordering of DataDriven Tests:
Xray-App/xray-testng-extensions#12

This is the test that was failing due to flakiness
https://github.com/Xray-App/xray-testng-extensions/blob/828fe56cb406e66b2aa2be1958fcd19ff17ba578/src/test/java/app/getxray/xray/testng/tests/XrayJsonReportTests.java#L197

[krmahadevan]

@itkhanz - Thanks for getting back. But I would still like to see a sample that can be used to reproduce the problem. So if you or @bitcoder can help share a test (or) alter the one that I shared earlier, I would still want to investigate as to what is going wrong.

[itkhanz]

This is the test case that was failing earlier because it expect the response in TestNG report to be in same order as the parameters in data driven scenario, and sometimes the order of iterations were different making the build flaky. We fixed it at our end by sorting the iterations and then making comparison.
https://github.com/Xray-App/xray-testng-extensions/blob/master/src/test/java/app/getxray/xray/testng/tests/XrayJsonReportTests.java#L195

and this is the data-driven scenario class and method that is being run
https://github.com/Xray-App/xray-testng-extensions/blob/828fe56cb406e66b2aa2be1958fcd19ff17ba578/src/test/java/app/getxray/xray/testng/tests/DataDrivenExamples.java#L26C1-L42C6

This is how the test class and method name are then being passsed to run the test:
https://github.com/Xray-App/xray-testng-extensions/blob/828fe56cb406e66b2aa2be1958fcd19ff17ba578/src/test/java/app/getxray/xray/testng/tests/XrayJsonReportTests.java#L509

[krmahadevan]

@itkhanz - The root cause of the issue is NOT how TestNG runs the tests but its related to how TestNG captures the results.

I see that the code does use a LinkedHashSet to sort the ITestResult objects based on the start time

But this is being persisted into results which is declared as a HashMap instead of it being a LinkedHashMap. So the ordering is being lost. See here

I think this code block could have been just replaced with something like this (which would ensure that the ordering is going to be maintained)

Map<String, List<ITestResult>> results = testResults.stream()
        .collect(
                Collectors.groupingBy(
                        itr -> itr.getMethod().getQualifiedName(),
                        LinkedHashMap::new,
                        Collectors.toList()
                )
        );

This will ensure that the entries getting added into results maintain the insertion order. So when you iterate this map and then add them into the JSONArray, the order in which they were found should be preserved.

results.values().forEach(it -> addTestResults(tests, it));

In a nutshell, I think your tests were right and they were actually hinting at a problem in the listener code which should have been fixed 😬

[krmahadevan]

@bitcoder @itkhanz - Let me know if that provides a convincing explanation to what could be going wrong in this case.

[bitcoder]

thanks @krmahadevan for your analysis and your time; some time ago I thought also the same (that it was due to the current listener implementation); I will have a look at it whenever I have a bit and I'll let you know