documentation incorrect in AfterTest.java

[jjfrankovich]

TestNG Version

Note: only the latest version is supported
latest

Expected behavior

The documentation for alwaysRun should be corrected to show that alwaysRun is not required to run cleanup afer a failed test.

 /**
   * For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not
   * beforeGroups): if set to true, this configuration method will be run regardless of what groups
   * it belongs to. <br>
   * For after methods (afterSuite, afterClass, ...): if set to true, this configuration method will
   * be run even if the group does not match. alwaysRun=true is no longer required if a test fails, just a matching group is sufficient.
   *
   * @return the value (default false)
   */
  boolean alwaysRun() default false;

Actual behavior

https://github.com/testng-team/testng/blob/master/testng-core-api/src/main/java/org/testng/annotations/AfterTest.java

 /**
   * For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not
   * beforeGroups): if set to true, this configuration method will be run regardless of what groups
   * it belongs to. <br>
   * For after methods (afterSuite, afterClass, ...): if set to true, this configuration method will
   * be run even if one or more methods invoked previously failed or was skipped.
   *
   * @return the value (default false)
   */
  boolean alwaysRun() default false;

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• [x ] IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

  @BeforeTest
  public void beforeTest()
  {
    System.out.println("beforeTest");
  }

  @AfterTest
  public void afterTest()
  {
    System.out.println("afterTest");
  }

  public void failedAssert()
  {
    assertEquals(true, false);
  }

  public void failedAssertWithThrow()
  {
    new RuntimeException("test");
  }

Results:

d:\niagara\r414>test baja:HtmlUtilsTest.failedAssert

beforeTest
afterTest

*** TestNG Failure Summary ***
 TestNG Test Failed: failedAssert on Html Utils Test - expected [false] but found [true]

===============================================
bajaTest_HtmlUtilsTest
Total tests run: 1, Failures: 1, Skips: 0
===============================================

===============================================
Results of all run tests
Total tests run: 1, Failures: 1, Skips: 0
===============================================

d:\>test baja:HtmlUtilsTest.failedAssertWithThrow
...
beforeTest
afterTest

===============================================
bajaTest_HtmlUtilsTest
Total tests run: 1, Failures: 0, Skips: 0
===============================================

===============================================
Results of all run tests
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[jjfrankovich]

I was just informed that alwaysRun applies to previous AfterXXX method (and not test method failures), so it would be good to clarify that in the javadoc:

For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not
beforeGroups): if set to true, this configuration method will be run regardless of what groups
it belongs to.
For after methods (afterSuite, afterClass, ...): if set to true, this configuration method will
be run even if a previous after method failed.
@return the value (default false)

[juherr]

Thanks for the report. The way you run the test may have impacts.
How can we reproduce the behavior? I mean: what is the test command doing?