Issue1991: DataProvider Execution
Bruce Wen edited this page Jan 3, 2019
·
7 revisions
This wiki page is to explain the Issue 1991.
- https://github.com/wengm/testng-training/blob/master/basic/src/test/java/org/testng/forum/i1991/DataProviderTest.java
- https://github.com/wengm/testng-training/blob/master/basic/src/suites/forum/i1991.xml
- https://github.com/wengm/testng-training/blob/master/basic/src/test/java/org/testng/forum/RunTestNg.java
TestNG:7.0.0-beta1 org.testng.SkipException: BeforeMethod: Skip Color:Red at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:511) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:241) at org.testng.internal.Invoker.invokeMethod(Invoker.java:581) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:792) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1103) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122) at org.testng.TestRunner.privateRun(TestRunner.java:739) at org.testng.TestRunner.run(TestRunner.java:589) at org.testng.SuiteRunner.runTest(SuiteRunner.java:398) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:354) at org.testng.SuiteRunner.run(SuiteRunner.java:302) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1145) at org.testng.TestNG.runSuitesLocally(TestNG.java:1067) at org.testng.TestNG.runSuites(TestNG.java:997) at org.testng.TestNG.run(TestNG.java:965) at org.testng.TestNG.privateMain(TestNG.java:1291) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) BeforeMethod: Not Skipped Color:White BeforeMethod: Not Skipped Color:Black =============================================== DataProviderTestSuite Total tests run: 3, Passes: 0, Failures: 0, Skips: 3 Configuration Failures: 0, Skips: 1 =============================================== BeforeMethod: Not Skipped Color:Red Test Method: show(String color): Red -> class org.testng.forum.i1991.DataProviderTest's instance:1995619265 org.testng.SkipException: BeforeMethod: Skip Color:White at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:511) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:241) at org.testng.internal.Invoker.invokeMethod(Invoker.java:581) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:792) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1103) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122) at org.testng.TestRunner.privateRun(TestRunner.java:739) at org.testng.TestRunner.run(TestRunner.java:589) at org.testng.SuiteRunner.runTest(SuiteRunner.java:398) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:354) at org.testng.SuiteRunner.run(SuiteRunner.java:302) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1145) at org.testng.TestNG.runSuitesLocally(TestNG.java:1067) at org.testng.TestNG.runSuites(TestNG.java:997) at org.testng.TestNG.run(TestNG.java:965) at org.testng.TestNG.privateMain(TestNG.java:1291) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) BeforeMethod: Not Skipped Color:Black =============================================== DataProviderTestSuite Total tests run: 3, Passes: 1, Failures: 0, Skips: 2 Configuration Failures: 0, Skips: 1 =============================================== BeforeMethod: Not Skipped Color:Red Test Method: show(String color): Red -> class org.testng.forum.i1991.DataProviderTest's instance:888655833 BeforeMethod: Not Skipped Color:White Test Method: show(String color): White -> class org.testng.forum.i1991.DataProviderTest's instance:888655833 org.testng.SkipException: BeforeMethod: Skip Color:Black at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:511) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:241) at org.testng.internal.Invoker.invokeMethod(Invoker.java:581) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:792) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1103) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:140) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122) at org.testng.TestRunner.privateRun(TestRunner.java:739) at org.testng.TestRunner.run(TestRunner.java:589) at org.testng.SuiteRunner.runTest(SuiteRunner.java:398) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:354) at org.testng.SuiteRunner.run(SuiteRunner.java:302) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1145) at org.testng.TestNG.runSuitesLocally(TestNG.java:1067) at org.testng.TestNG.runSuites(TestNG.java:997) at org.testng.TestNG.run(TestNG.java:965) at org.testng.TestNG.privateMain(TestNG.java:1291) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) =============================================== DataProviderTestSuite Total tests run: 3, Passes: 2, Failures: 0, Skips: 1 Configuration Failures: 0, Skips: 1 ===============================================
TestNG:6.14.3 org.testng.SkipException: BeforeMethod: Skip Color:Red at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222) at org.testng.internal.Invoker.invokeMethod(Invoker.java:523) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.privateRun(TestRunner.java:648) at org.testng.TestRunner.run(TestRunner.java:505) at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415) at org.testng.SuiteRunner.run(SuiteRunner.java:364) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208) at org.testng.TestNG.runSuitesLocally(TestNG.java:1137) at org.testng.TestNG.runSuites(TestNG.java:1049) at org.testng.TestNG.run(TestNG.java:1017) at org.testng.TestNG.privateMain(TestNG.java:1354) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) BeforeMethod: Not Skipped Color:White BeforeMethod: Not Skipped Color:Black =============================================== DataProviderTestSuite Total tests run: 3, Failures: 0, Skips: 3 Configuration Failures: 0, Skips: 1 =============================================== BeforeMethod: Not Skipped Color:Red Test Method: show(String color): Red -> class org.testng.forum.i1991.DataProviderTest's instance:324457684 org.testng.SkipException: BeforeMethod: Skip Color:White at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222) at org.testng.internal.Invoker.invokeMethod(Invoker.java:523) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.privateRun(TestRunner.java:648) at org.testng.TestRunner.run(TestRunner.java:505) at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415) at org.testng.SuiteRunner.run(SuiteRunner.java:364) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208) at org.testng.TestNG.runSuitesLocally(TestNG.java:1137) at org.testng.TestNG.runSuites(TestNG.java:1049) at org.testng.TestNG.run(TestNG.java:1017) at org.testng.TestNG.privateMain(TestNG.java:1354) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) BeforeMethod: Not Skipped Color:Black =============================================== DataProviderTestSuite Total tests run: 3, Failures: 0, Skips: 2 Configuration Failures: 0, Skips: 1 =============================================== BeforeMethod: Not Skipped Color:Red Test Method: show(String color): Red -> class org.testng.forum.i1991.DataProviderTest's instance:1997353766 BeforeMethod: Not Skipped Color:White Test Method: show(String color): White -> class org.testng.forum.i1991.DataProviderTest's instance:1997353766 org.testng.SkipException: BeforeMethod: Skip Color:Black at org.testng.forum.i1991.DataProviderTest.beforeMethod(DataProviderTest.java:25) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:458) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222) at org.testng.internal.Invoker.invokeMethod(Invoker.java:523) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.privateRun(TestRunner.java:648) at org.testng.TestRunner.run(TestRunner.java:505) at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415) at org.testng.SuiteRunner.run(SuiteRunner.java:364) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208) at org.testng.TestNG.runSuitesLocally(TestNG.java:1137) at org.testng.TestNG.runSuites(TestNG.java:1049) at org.testng.TestNG.run(TestNG.java:1017) at org.testng.TestNG.privateMain(TestNG.java:1354) at org.testng.forum.RunTestNg.i1991(RunTestNg.java:19) at org.testng.forum.RunTestNg.main(RunTestNg.java:11) =============================================== DataProviderTestSuite Total tests run: 3, Failures: 0, Skips: 1 Configuration Failures: 0, Skips: 1 ===============================================
Note:
- Only one instance of test class is created and used for multiple test run for different data groups.
- @BeforeMethod is executed for each test run for each data group.
- Skipped @BeforeMethod in one test run will impact the remaining test runs.
- PASSED test run is not displayed in 6.14.3
Summary:
- If @BeforeMethod is executed for each test run for each data group, then the test run should be atomic and @BeforeMethod of one test run should not impact the other test runs.
- From concept perspective, @BeforeMethod should be executed before test method rather than each test run for each data group when data provider is used.