Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file #2731

Open
1 of 7 tasks
Mei-self opened this issue Feb 17, 2022 · 6 comments · May be fixed by #2735
Open
1 of 7 tasks

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file #2731

Mei-self opened this issue Feb 17, 2022 · 6 comments · May be fixed by #2735
Labels
Feature: configfailurepolicy

Comments

@Mei-self
Copy link

Mei-self commented Feb 17, 2022 •
edited

TestNG Version

7.5

Expected behavior

if configfailurepolicy=continue and any BeforeXXX method fails, the remaining test method should execute not skip

Actual behavior

only BeforeTest fails, the remaining test method would execute

Is the issue reproducible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestSuite" configfailurepolicy="continue">
    <test name="Test">
        <classes>
            <class name="TestConfigFail"/>
        </classes>
    </test>
</suite>

public class TestConfigFail {

    @BeforeSuite
    public void beforeSuite() {
        Assert.fail("This before suite is fail");
    }

    @BeforeTest
    public void beforeTest() {
        Assert.fail("This before test is fail");
    }

    @BeforeClass
    public void beforeClass(){
        Assert.fail("This before class is fail");
    }

    @BeforeMethod
    public void beforeMethod(){
        Assert.fail("This before method is fail");
    }

    @Test
    public void test() {
        System.out.println("This is test method");
    }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.
@krmahadevan
Copy link
Member

@Wang-Meimei - Is this something that was broken in 7.5 or has this never worked ? Can u please check in 7.4.0 and 6.14.3 and post back your findings ?

@Mei-self
Copy link
Author

@Wang-Meimei - Is this something that was broken in 7.5 or has this never worked ? Can u please check in 7.4.0 and 6.14.3 and post back your findings ?

It seems this never worked. But from the document, it should work. So maybe need to update document?

@juherr
Copy link
Member

juherr commented Feb 18, 2022

It seems this never worked. But from the document, it should work. So maybe need to update document?

Yes, as we know now it is no a regression, we need to determine the expected behavior and fix the code or the doc.

I think the issue is valid and the method should be run.
@krmahadevan wdyt?

bj-9527 added a commit to bj-9527/testng that referenced this issue Feb 24, 2022
@bj-9527
configfailurepolicy=continue only works for BeforeTest when using Tes…
aafb2d1 
…tNG XML file

Fixes testng-team#2731
@bj-9527 bj-9527 linked a pull request Feb 24, 2022 that will close this issue
Open
3 tasks
@krmahadevan
Copy link
Member

It seems this never worked. But from the document, it should work. So maybe need to update document?

Yes, as we know now it is no a regression, we need to determine the expected behavior and fix the code or the doc.

I think the issue is valid and the method should be run. @krmahadevan wdyt?

I dont know if this is an issue @juherr .

If there is a higher and lower order configuration, then we should not try to execute the lower order configuration.

For .e.g,

Lets say I have

  • @BeforeSuite
  • @BeforeTest
  • @Test

In this case, we should not try to execute the @BeforeTest because this is a configuration method. But then one can argue that if one has to get to the level of running a @Test method, then all the above level configurations need to be run.

So am not sure what the behaviour should be.

@bj-9527 - Please note.

@juherr
Copy link
Member

juherr commented Feb 28, 2022

IMO, the only difference between continue and skip is that continue will execute the method, and skip won't execute and will directly set the SKIP status.

But when I see #2735 changes in tests, I wonder if the behavior change should not be configurable.

@bj-9527
Copy link
Contributor

bj-9527 commented Mar 1, 2022

It could be config by FailurePolicy which could be set as SKIP or CONTINUE, then the behavior could change.

So am not sure what the behaviour should be.

For, this change, the SKIP are working as before, the CONTINUE will execute the following method even previous config failed.

bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
@bj-9527
configfailurepolicy=continue only works for BeforeTest when using Tes…
a30dc42 
…tNG XML file

Fixes testng-team#2731
bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
@bj-9527
configfailurepolicy=continue only works for BeforeTest when using Tes…
86d924d 
…tNG XML file

Fixes testng-team#2731
bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
@bj-9527
configfailurepolicy=continue only works for BeforeTest when using Tes…
2f4b7c7 
…tNG XML file

Fixes testng-team#2731
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature: configfailurepolicy
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file bj-9527/testng
4 participants
@juherr @krmahadevan @bj-9527 @Mei-self