Before/After methods are not running when groups "include" in suite #1574

chetand24 · 2017-10-13T17:57:07Z

TestNG Version : 6.12.0, 6.11.0

Note: only the latest version is supported

Expected behavior

The Before and After methods should run when "include" tag is present in the testng suite

Actual behavior

The Before and After methods are not running when "include" tag present in the testng suite. But is working fine when "exclude" is present

Is the issue reproductible on runner?

Shell
Maven
[* ] Gradle
Ant
[ *] Eclipse
[ *] IntelliJ
NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

The test class:

public class AnotherTest {

    @BeforeSuite
    public void setupSuite(){
        System.out.println("Before Suite Setup ...... ");
    }

    @BeforeTest
    public void setUpTest() throws Exception {
        System.out.println("Before Test Setup ...... ");
    }

    @BeforeMethod
    public void setUpMethod() throws URISyntaxException {
        System.out.println("Before Method Setup ...... ");
    }

    @AfterMethod
    public void tearDownMethod(){
        System.out.println("After Method Setup ...... ");
    }

    @AfterTest
    public void tearDownSuite() throws Exception {
        System.out.println("After test Setup ...... ");
    }

    @Test(groups = {"sometest"})
    public void someTest(){
        System.out.println("Some test ...... ");
    }

    @Test(groups = {"anothertest"})
    public void anothertest(){
        System.out.println("Another test ...... ");
    }
}

Example XML Suite

<suite name="Mobile Test" verbose="10">
    <test name="Some Tests">
        <groups>
            <run>
                <!--<exclude name="sometest"/>-->
                <include name="sometest"/>
            </run>
        </groups>
        <classes>
            <class name="com.practice.AnotherTest"/>
        </classes>
    </test>
</suite>

Output when included the group:

...
... TestNG 6.11 by Cédric Beust (cedric@beust.com)
...
Some test ...... 
===== Invoked methods
    AnotherTest.someTest()[pri:0, instance:com.practice.AnotherTest@7cf10a6f] 2096171631
=====
===============================================
Mobile Automated Test for Engage Android
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Output when excluded the group:

...
... TestNG 6.11 by Cédric Beust (cedric@beust.com)
...
Before Suite Setup ...... 
Before Test Setup ...... 
Before Method Setup ...... 
Another test ...... 
After Method Setup ...... 
After test Setup ...... 
===== Invoked methods
  AnotherTest.setupSuite()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.setUpTest()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.setUpMethod()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
    AnotherTest.anothertest()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.tearDownMethod()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.tearDownSuite()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
=====

===============================================
Mobile Automated Test for Engage Android
Total tests run: 1, Failures: 0, Skips: 0
===============================================

The text was updated successfully, but these errors were encountered:

Closes testng-team#1574 Suppose there are one or more configuration methods which don’t belong to any group and for which “alwaysRun” attribute is not set, and when the user specifies a valid exclusion list for groups, TestNG would still execute those configuration methods. Fixed this anomaly.

krmahadevan · 2017-10-14T09:14:40Z

@chetand24 - There are two parts to your issue.

The Before and After methods are not running when "include" tag present in the testng suite.

By default if you would like your BeforeXXX and AfterXXX to be executed all the time, irrespective of what group is being executed, you should be using setting : alwaysRun=true

But is working fine when "exclude" is present

This part is a bug. I have fixed this, by raising a pull request ( #1575 )

Closes testng-team#1574 Suppose there are one or more configuration methods which don’t belong to any group and for which “alwaysRun” attribute is not set, and when the user specifies a valid exclusion list for groups, TestNG would still execute those configuration methods. Fixed this anomaly.

juherr · 2017-10-15T18:59:44Z

@krmahadevan I'm not sure if it is a bug. The documentation doesn't specify it, but we can imagine that no group means the "default group".

So, include "x" won't add "default group" but exclude "x" won't remove "default group".

@cbeust Could you clarify the expected behavior of "exclude" for methods without a group?

juherr · 2017-10-15T19:03:41Z

In fact, if we remove "no group" methods with "exclude", I don't know how to run "no group" and some other groups methods.

krmahadevan · 2017-10-16T03:48:51Z

@juherr - You are right in terms of the behavior not defined properly.

Here's my understanding :

When a user specifies run by groups, then one explicitly likes to run one or more groups.
A test/config method without any groups, means its explicitly meant NOT to be run via the groups selection mechanism.
There is no concept of default/default group groups (similar to how we have a default package when we don't specify the package name for a class) in TestNG.
Currently no group (or) default group with include doesn't work any way because a user cannot combine methods with groups specified and methods with no groups specified together in the single execution via groups mechanism. Not at-least with the @Test methods. So the behavior should be consistent for exclude as well.

Closes testng-team#1574 Suppose there are one or more configuration methods which don’t belong to any group and for which “alwaysRun” attribute is not set, and when the user specifies a valid exclusion list for groups, TestNG would still execute those configuration methods. Fixed this anomaly.

kool79 · 2017-11-21T22:54:32Z

@krmahadevan, your PR will broke backward compatibility. In my projects I use groups feature only to exclude tests (flacky, app-bug, time-consumind, etc). So I mark only those tests which I want to exclude. After your PR all my tests without groups will be excluded from run. I'll be forced to assign some groups to ALL methods when I want to exclude just 1 test-case. This is not obvious (I need to assign ANY group to method if I want to run it with exclusions activated)
I'll try to rectify your understanding:

When you need to run only certain tests/methods, define group(s) for them
When you need to exclude certain tests/methods, define group(s) for them

About

Currently no group (or) default group with include doesn't work any way because a user cannot combine methods with groups specified and methods with no groups specified together in the single execution via groups mechanism.

Test with no-group belongs to virtual 'all-tests' group (as same as tests with groups). No need to include something into this group because it already contains all stuffs. But if you still want to combine (no-group) and some-group then probably you just need one more group like common-tests for such tests.
P.S. Of cause, you always can write custom filter to override default behavior of XmlMethodSelector

krmahadevan · 2017-11-22T03:32:51Z

@kool79 - Thanks for your inputs. The only reason why this PR is pending merge and will perhaps never be merged is because of backward compatibility. The current behavior IMHO is not intuitive.
Methods which dont belong to a group, mean they dont belong to a group. There cannot be a virtual all-tests group to which they get automagically added.

But debates apart, backward compatibility is more important than anything else. So this PR will not be merged.

javydreamercsw · 2020-04-30T15:31:10Z

It almost begs the question if alwaysRun should default to true and be disabled if the method has a group.

krmahadevan linked a pull request Oct 14, 2017 that will close this issue

After methods should be skipped when exclusion used #1575

Open

2 tasks

krmahadevan self-assigned this Oct 14, 2017

juherr mentioned this issue Nov 27, 2017

BUG: Parameter alwaysRun=true for before-methods forces execution of those methods #1622

Open

7 tasks

cjayswal mentioned this issue Mar 13, 2019

When groups "include" in suite, ResultUpdator not awaiting for completion qmetry/qaf#249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Before/After methods are not running when groups "include" in suite #1574

Before/After methods are not running when groups "include" in suite #1574

chetand24 commented Oct 13, 2017 •

edited by krmahadevan

krmahadevan commented Oct 14, 2017

juherr commented Oct 15, 2017

juherr commented Oct 15, 2017

krmahadevan commented Oct 16, 2017

kool79 commented Nov 21, 2017

krmahadevan commented Nov 22, 2017

javydreamercsw commented Apr 30, 2020

Before/After methods are not running when groups "include" in suite #1574

Before/After methods are not running when groups "include" in suite #1574

Comments

chetand24 commented Oct 13, 2017 • edited by krmahadevan

TestNG Version : 6.12.0, 6.11.0

Expected behavior

Actual behavior

Is the issue reproductible on runner?

Test case sample

krmahadevan commented Oct 14, 2017

juherr commented Oct 15, 2017

juherr commented Oct 15, 2017

krmahadevan commented Oct 16, 2017

kool79 commented Nov 21, 2017

krmahadevan commented Nov 22, 2017

javydreamercsw commented Apr 30, 2020

chetand24 commented Oct 13, 2017 •

edited by krmahadevan