Feature request: Add new fail config failure policy

[pavetok]

TestNG Version

LATEST

Problem

Can't to fail fast CI build right after configuration method failure.

Maven Surefire has option skipAfterFailureCount. It allows us to fail E2E tests build after first test method failure. So we can investigate issue in relevant stand state. But we can't achieve the same for configuration method failures.

Solution

Add new fail config failure policy. So all test methods in test class would be failed and CI build would be failed asap too.

[krmahadevan]

@pavetok - TestNG is already returning back a return code of 1 when there are configuration failures. Have you tried running a test at the command prompt and then printing the last return code using echo $? ? You should see a non negative value. Not sure what additional can be done here in terms of failure policy, when its already being done by TestNG.
Maybe you may have to look at your CI system to find out what may be going wrong ?

[krmahadevan]

@pavetok - If you are using Jenkins, you should perhaps check and ensure that the recommendation in this stackoverflow thread is done.

Add -Dmaven.test.failure.ignore=false to MAVEN_OPTS.

[pavetok]

If you are using Jenkins, you should perhaps check and ensure that the recommendation in this stackoverflow thread is done.

@krmahadevan - We are using TeamCity and by default this option disabled in the surefire plugin https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore

[pavetok]

TestNG is already returning back a return code of 1 when there are configuration failures.

@krmahadevan - Code of 1 will be returned after process exit at the very end of test run. It seems it can't be used for the purpose of a fast fail.

[krmahadevan]

@krmahadevan - We are using TeamCity and by default this option disabled in the surefire plugin

If you read the link that I shared, it also talks about the same thing but this gets enabled by Jenkins. Maybe TeamCity is doing the same thing too? Have you tried setting it and then observing if there is any difference in behavior ?

@krmahadevan - Code of 1 will be returned after process exit at the very end of test run. It seems it can't be used for the purpose of a fast fail.

The exit will depend on which configuration had a failure. If it was a @BeforeSuite it would be immediate. If it was a @BeforeTest that particular <test> will be skipped, but TestNG will proceed to execute the next <test> tag and so on for @BeforeClass etc.,

If you would want to abort the execution from moving further after any failure, I think you can do the same with a TestNG listener, but throwing a SkipException