TestNG fails to finish gracefully when executing tests with the dependsOnMethods attribute if an exception happens in ITestListener.onTestStart #2960
Comments
harmin-parra
changed the title
harmin-parra
changed the title
|
There is no expected exception in listener interfaces: you are supposed to manage exceptions in the listener implementation. But TestNG can do better and the expected behavior from listeners should be the same as through annotated methods. Could you check and tell us what is happening when a |
|
The exception seems to be well handled in the I observe the same behavior and outcome. No test result summary at the end of the execution. |
|
@harmin-parra - Listeners and configuration methods are IMO two different things. While a listener can also be made to work as if it were a configuration method, I dont think that listeners should be perceived that way. Listeners are essentially supposed to work as if they were call backs which lets you run some additional code before/after something happens. Configuration methods are supposed to be a subset of these callback functionalities that listener provides and are strictly confined to the lifecycle of tests. I say this because, currently we can have a listener to report test reports, but there's no configuration method similar to that. We can have a listener that can be configured to alter a suite (via So not sure if we can look at both of them that way. That perhaps is why TestNG does not handle any exceptions that can arise out of the listeners. @juherr - What do you suggest should be the further course of action on this? I feel that we just need to make it explicit in documentation that listeners are not supposed to throw exceptions and they should be handling them on their own. |
|
Yes. The documentation should be updated. But I don't see good reason that explains the failure here. |
|
@juherr - The listener is explicitly throwing an exception See here https://github.com/harmin-parra/issue-2960/blob/master/src/TestListener.java#L14 That perhaps explains why the failure occurs Can be easily simulated using the below sample import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners(SampleTestClass.TestListener.class)
public class SampleTestClass {
@Test
public void testMethod() {
}
@Test(dependsOnMethods = "testMethod")
public void anotherTestMethod() {
}
public static class TestListener implements ITestListener {
@Override
public void onTestStart(ITestResult result) {
throw new SkipException("Don't run");
}
}
} |
|
Execution logs look like below [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project testng_playground:
[ERROR]
[ERROR] Please refer to /Users/kmahadevan/githome/playground/testng_playground/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] There was an error in the forked process
[ERROR] Don't run
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] Don't run
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
[ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
[ERROR] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
[ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
➜ testng_playground
|
|
Sure but I remember we catched exceptions from listener with a warning. |
|
Do you remember for which listener we were catching exceptions? |
|
Not sure but I think it was before/afterMethod |
|
The reason I am throwing an exception in a Listener is because I want to be able to skip the remaining tests of a suite if a test fails. And I don't want to use the I guess this issue could be rejected if you agree to an evolution and decide provide a command-line parameter option called |
|
@harmin-parra - That's a good suggestion. The only challenge with that would be to determine the failure strategy when On a side note, what you are asking for, can be accomplished with TestNG using Test classes would look like below package com.rationaleemotions;
import org.testng.Assert;
import org.testng.annotations.Test;
public class FirstClassTest {
@Test
public void aTest1() {}
@Test
public void failingTest() {
Assert.fail("I am to be failing");
}
}package com.rationaleemotions;
import org.testng.Assert;
import org.testng.annotations.Test;
public class SecondClassTest {
@Test
public void skippingTest1() {
Assert.fail("I should NOT have been executed");
}
@Test
public void skippingTest2() {
Assert.fail("I should NOT have been executed");
}
}The sample listener which manages the package com.rationaleemotions;
import org.testng.*;
import org.testng.xml.XmlSuite;
import java.util.List;
import java.util.Set;
public class FailFastListener implements ITestListener, IReporter {
@Override
public void onTestStart(ITestResult result) {
if (result.getTestContext().getFailedTests().size() > 0) {
result.setStatus(ITestResult.SKIP);
result.setThrowable(new SkipException("Skipping this test"));
}
}
@Override
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
suites.stream()
.flatMap(each -> each.getResults().values().stream())
.map(ISuiteResult::getTestContext)
.forEach(each -> {
print("Passed Tests", each.getPassedTests().getAllResults());
print("Failed Tests", each.getFailedTests().getAllResults());
print("Skipped Tests", each.getSkippedTests().getAllResults());
});
}
private static void print(String msg, Set<ITestResult> results) {
System.err.println(msg);
results.forEach(each -> System.err.println(each.getMethod().getQualifiedName()));
}
}Here's a suite file that uses all of this <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Practice Suite" verbose="2">
<listeners>
<listener class-name="com.rationaleemotions.FailFastListener"/>
</listeners>
<test name="Test Basics 1" verbose="2" parallel="false">
<classes>
<class name="com.rationaleemotions.FirstClassTest"/>
<class name="com.rationaleemotions.SecondClassTest"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->Here's the execution output ===============================================
Test Basics 1
Tests run: 4, Failures: 1, Skips: 2
===============================================
Passed Tests
com.rationaleemotions.FirstClassTest.aTest1
Failed Tests
com.rationaleemotions.FirstClassTest.failingTest
Skipped Tests
com.rationaleemotions.SecondClassTest.skippingTest2
com.rationaleemotions.SecondClassTest.skippingTest1
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 0.487 s <<< FAILURE! -- in TestSuite
[ERROR] com.rationaleemotions.FirstClassTest.failingTest -- Time elapsed: 0.003 s <<< FAILURE!
java.lang.AssertionError: I am to be failing
at org.testng.Assert.fail(Assert.java:111)
at com.rationaleemotions.FirstClassTest.failingTest(FirstClassTest.java:14)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:664)
at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:227)
at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:957)
at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:200)
at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.testng.TestRunner.privateRun(TestRunner.java:848)
at org.testng.TestRunner.run(TestRunner.java:621)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:443)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:437)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:397)
at org.testng.SuiteRunner.run(SuiteRunner.java:336)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1280)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1200)
at org.testng.TestNG.runSuites(TestNG.java:1114)
at org.testng.TestNG.run(TestNG.java:1082)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:308)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:71)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:113)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] FirstClassTest.failingTest:14 I am to be failing
[INFO]
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 2
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.982 s
[INFO] Finished at: 2023-09-10T19:41:04+05:30 |
TestNG Version
Description
If I have a test annotation with the
dependsOnMethodsattribute, and if an exception occurs inITestListener.onTestStart, TestNG is unable to finish the test execution and exists in an error state.ITestListener.onTestSkippednever gets called for the test with thedependsOnMethodsattribute.The test result summary never gets displayed in the console.
Expected behavior
TestNG executes and displays the test result summary in the console.
'onTestSkipped' gets printed twice.
Actual behavior
TestNG doesn't execute until the end.
The test result summary never gets displayed.
"onTestSkipped" gets printed once.
Is the issue reproducible on runner?
Test case sample
See repo with sample code
https://github.com/harmin-parra/issue-2960
Info
If I replace
@Test(dependsOnMethods = {"test1"})by@Test, TesnNG executes until the end and I get the test result summaryThe text was updated successfully, but these errors were encountered: