Test run wrong order when use DependsOnGroups and Parallel by classes

[hieuhth]

TestNG Version

7.8.x

Expected behavior

There are 3 classes are belong to group "First", and 3 classes are belong to group "Second". Group "Second" depends on group "First".

In test suite, Config to run parallel by classes, thread-count=6.
3 classes of group "First" will run parallel. After they run completely, 3 classes of group "Second" should be run parallel.

Actual behavior

3 classes of group "First" will run parallel. After they run completely, 3 classes of group "Second" run one by one, not parallel

Is the issue reproducible on runner?

There are 3 class of group "First"

public class FirstTest1 {
    @Test(groups = "First")
    public void FirstTest1() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest1 " + new Date());
    }
}

public class FirstTest2 {
    @Test(groups = "First")
    public void FirstTest2() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest2 " + new Date());
    }
}

public class FirstTest3 {
    @Test(groups = "First")
    public void FirstTest3() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest3 " + new Date());
    }
}

And 3 classes of group "Second"

public class SecondTest1 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest1() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest1 " + new Date());
    }
}

public class SecondTest2 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest2() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest2 " + new Date());
    }
}

public class SecondTest3 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest3() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest3 " + new Date());
    }
}

Test suite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Suite" parallel="classes" thread-count="10">
    <test name="Tests">
        <parameter name="browserValue" value="chrome"/>
        <classes>
            <class name="com.simplicia.test.FirstTest1"></class>
            <class name="com.simplicia.test.FirstTest2"></class>
            <class name="com.simplicia.test.FirstTest3"></class>
            <class name="com.simplicia.test.SecondTest1"></class>
            <class name="com.simplicia.test.SecondTest2"></class>
            <class name="com.simplicia.test.SecondTest3"></class>
        </classes>
    </test>
</suite>

Run test suite,

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[juherr]

Related to #2372

[krmahadevan]

@hieuhth - What version of TestNG are you working with ?

I tried using 7.8.0 and I am not able to reproduce the issue. Please try using the latest version and post back your findings

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
...
... TestNG 7.8.0 by Cédric Beust (cedric@beust.com)
...

Run FirstTest3 Mon Sep 25 16:58:38 IST 2023
Run FirstTest2 Mon Sep 25 16:58:38 IST 2023
Run FirstTest1 Mon Sep 25 16:58:38 IST 2023
Run SecondTest2 Mon Sep 25 16:58:43 IST 2023
Run SecondTest1 Mon Sep 25 16:58:43 IST 2023
Run SecondTest3 Mon Sep 25 16:58:43 IST 2023

===============================================
Test Suite
Total tests run: 6, Passes: 6, Failures: 0, Skips: 0
===============================================

[hieuhth]

@krmahadevan I use testng 7.8.0.

I have just rechecked, I set JAVA_TOOL_OPTIONS -Dtestng.thread.affinity=true (to resolve a other issue in my project).

If I remove this config, issue will not occur.
How to use this config without issue which I posted?

[krmahadevan]

Great. Then i guess we can close this issue @hieuhth ?

[hieuhth]

@krmahadevan I don't think so.

testng.thread.affinity : resort to running dependent methods on the same thread as the upstream method

-> I think we should only resort methods of same class when run parallel by classes, and depend is used.

[krmahadevan]

@hieuhth

testng.thread.affinity : resort to running dependent methods on the same thread as the upstream method

A couple of things about this feature

• This is an experimental feature.
• This works ONLY when you have EXACTLY 1 upstream method.

-> I think we should only resort methods of same class when run parallel by classes, and depend is used.

I don't think that's possible. It can be done in a limited fashion when you use the above mentioned JVM argument, but this cannot be used as a generalised strategy. I say this because upstream can always be more than 1 method and so its not going to be possible to figure out on which thread to run a dependent method, when it has more than 1 upstream dependencies.

I would suggest that you relook at your tests. Having a thread affinity in tests kind of sounds like a code smell. I am guessing you are resorting to this because you are having some sort of a dependency on a thread local variable. I would suggest that you re-look at your approach and either resort to building customisations that would ensure that the same thread local variable gets copied over to the dependent thread (or) you remove that dependency completely from your test.

The plain vanilla execution of dependsOngroups and parallel running works fine without any issues. The experimental thread affinity feature was never guaranteed to work for all use cases. It was meant to work ONLY for a specific use case, wherein you have a method that depends on exactly 1 upstream. Also the side effects of this behaviour when coupled with other hard ordering features (such as dependsOnGroups) is NOT documented.