[BUG]: Total test run duration/count is incorrectly calculated from junit xml report #19857

itkhanz · 2024-05-12T14:28:38Z

New issue checklist

Task name

Gradle@3 - Gradle v3 task

Task version

3

Issue Description

I am running my TestNG test suite in Azure pipelines via gradle task as gradle test. Since this task has by default publishJUnitResults property to true so it automatically publishes the test results after build. I am specifying the location of Junit XML reports generated by TestNG for this task to scan and then process test results. This is being done intentionally as be default gradle also produces Junit XML reports as **/TEST-*.xml which I have disabled to avoid test count duplication and pointed this task to load Junit XML reports generated by TestNG instead.

Although it correctly reads the test count and individual test durations but it shows incorrect total run duration as you can see in the screenshot below:

The correct test duratation is 8.26 minutes (sum of individual tests) as it can also be seen in the gradle logs, pipeline summary tab, as well as in the testng-results.xml file:

Job Summary

Gradle logs

2024-05-12T13:43:07.358+0000 [LIFECYCLE] [org.gradle.api.Task] 
2024-05-12T13:43:07.363+0000 [LIFECYCLE] [org.gradle.api.Task]   4 passing (8m 18s)
  3 failing

2024-05-12T13:43:07.369+0000 [QUIET] [system.out] 
2024-05-12T13:43:07.369+0000 [QUIET] [system.out] ---------------------------------------------------------------
2024-05-12T13:43:07.369+0000 [QUIET] [system.out] |  Results: FAILURE (7 tests, 4 passed, 3 failed, 0 skipped)  |
2024-05-12T13:43:07.369+0000 [QUIET] [system.out] ---------------------------------------------------------------
2024-05-12T13:43:07.369+0000 [DEBUG] [TestEventLogger] 
2024-05-12T13:43:07.370+0000 [DEBUG] [TestEventLogger] Gradle Test Run :test FAILED
2024-05-12T13:43:07.373+0000 [INFO] [org.gradle.api.Task] Test report disabled, omitting generation of the HTML test report.
2024-05-12T13:43:07.374+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute executeTests for :test' completed
2024-05-12T13:43:07.374+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Executing task ':test'' completed
2024-05-12T13:43:07.374+0000 [DEBUG] [org.gradle.internal.vfs.impl.AbstractVirtualFileSystem] Invalidating VFS paths: [/__w/1/s/build/allure-results, /__w/1/s/build/test-results/test/binary, /__w/1/s/build/testng, /__w/1/s/build/testng]
2024-05-12T13:43:07.383+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Snapshot outputs after executing task ':test'' completed
2024-05-12T13:43:07.387+0000 [DEBUG] [org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter] Removed task artifact state for task ':test' from context.
2024-05-12T13:43:07.387+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Task :test'
2024-05-12T13:43:07.387+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner2024-05-12T13:43:07.370+0000 [ERROR] [org.gradle.api.internal.tasks.testing.logging.TestCountLogger] 
7 tests completed, 3 failed
] Build operation 'Task :test' completed

testng-results.xml

Total test suite duration is 495755 ms = 8.26 minutes

<?xml version="1.0" encoding="UTF-8"?>
<testng-results ignored="0" total="7" passed="4" failed="3" skipped="0">
  <reporter-output>
  </reporter-output>
  <suite started-at="2024-05-12T13:34:51 UTC" name="Gradle suite" finished-at="2024-05-12T13:43:06 UTC" duration-ms="495755">
    <groups>
      <group name="b2b.smoketestFull">

Actual Result:
Total run duration is calculated only from the latest XML result file parsed. In this case, whatever is the last test suite (XML file) detected, it will become total test run duration. As the run duration of last read result is 2m 4s, hence the total run duration is shown as 2m 4s.

Expected Result:
Total run duration should be calculated as the sum of time durations of detected testsuites.

TestNG 7.10.1
Gradle 8.7

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

No response

Operation system

MacOS Sonoma 14.4.1

Relevant log output

##[debug]Object of TestRunPublisher created.
##[debug]Object of TestLogStore created.
##[debug]Failed to get FF TestManagement.Server.JUnitTestCaseAttachmentsEnabled Value.
##[debug]Object of JUnitResultParser created.
##[debug]Entering ParseTestResultFiles
##[debug]runContext.ReleaseURI is null
##[debug]runContext.ReleaseEnvironmentUri is null
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.Favourites.FavouritesTest.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 2.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.MapView.VSDS_2382_Check_Succesfully_Charging_Via_Selecting_Charge_Point_From_Map.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PoiDetailView.VSDS_2424_Verify_Redirection_to_Charging_Page_in_case_Charge_Point_has_both_RFID_And_App_Option.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Legal.VSDS_2376_Verify_that_Terms_and_Conditions_Are_Shown_Successfully.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Login.VSDS_2412_Verify_That_User_Can_Login_Successfully.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.RoutePlanner.VSDS_2397_VSDS_2399_Verify_Redirection_To_TripPlanner_And_Create_RoutePlan_Without_Stops_Then_Redirect_To_Map.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Leaving ParseTestResultFiles

Full task logs with system.debug enabled

 
##[debug]pattern: '/__w/1/s/**/build/testng/junitreports/TEST-*.xml'
##[debug]applying include pattern
##[debug]6 matches
##[debug]6 final results
##[debug]testRunTitle=undefined
##[debug]TestManagement.Server.UsePublishTestResultsLibInAgent is on
##[debug]TestManagement.Server.EnableAzureTestPlanTaskFlow is off
##[debug]TestManagement.Server.TriggerCoverageMergeJob is on
##[debug]Agent environment resources - Disk: / Available 87008.00 MB out of 99067.00 MB, Memory: Used 1620.00 MB out of 16009.00 MB, CPU: Usage 2.12%
##[debug]Object of TestLogStore created.
Result Attachments will be stored in LogStore
##[debug]Object of TestResultsPublisher created.
##[debug]Object of TestLogStore created.
Run Attachments will be stored in LogStore
##[debug]Object of TestRunPublisher created.
##[debug]Object of TestLogStore created.
##[debug]Failed to get FF TestManagement.Server.JUnitTestCaseAttachmentsEnabled Value.
##[debug]Object of JUnitResultParser created.
##[debug]Entering ParseTestResultFiles
##[debug]runContext.ReleaseURI is null
##[debug]runContext.ReleaseEnvironmentUri is null
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.Favourites.FavouritesTest.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 2.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.MapView.VSDS_2382_Check_Succesfully_Charging_Via_Selecting_Charge_Point_From_Map.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PoiDetailView.VSDS_2424_Verify_Redirection_to_Charging_Page_in_case_Charge_Point_has_both_RFID_And_App_Option.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Legal.VSDS_2376_Verify_that_Terms_and_Conditions_Are_Shown_Successfully.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Login.VSDS_2412_Verify_That_User_Can_Login_Successfully.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Reading test results from file '/agent/_work/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.RoutePlanner.VSDS_2397_VSDS_2399_Verify_Redirection_To_TripPlanner_And_Create_RoutePlan_Without_Stops_Then_Redirect_To_Map.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 1.
##[debug]Leaving ParseTestResultFiles
##[debug]runContext.ReleaseURI is null
##[debug]runContext.ReleaseEnvironmentUri is null
##[debug]Processed: ##vso[results.publish type=JUnit;mergeResults=true;publishRunAttachments=true;resultFiles=/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.Favourites.FavouritesTest.xml,/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.MapView.VSDS_2382_Check_Succesfully_Charging_Via_Selecting_Charge_Point_From_Map.xml,/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PoiDetailView.VSDS_2424_Verify_Redirection_to_Charging_Page_in_case_Charge_Point_has_both_RFID_And_App_Option.xml,/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Legal.VSDS_2376_Verify_that_Terms_and_Conditions_Are_Shown_Successfully.xml,/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.PrivateArea.Account.Login.VSDS_2412_Verify_That_User_Can_Login_Successfully.xml,/__w/1/s/build/testng/junitreports/TEST-com.dcs.appium.dcsapps.Tests.RoutePlanner.VSDS_2397_VSDS_2399_Verify_Redirection_To_TripPlanner_And_Create_RoutePlan_Without_Stops_Then_Redirect_To_Map.xml;testRunSystem=VSTS - gradle;]
##[debug]task result: Failed
##[debug]Entering PublishAllTestResultsToSingleTestRunAsync
##[error]Code analysis failed. Gradle exit code: -1. Error: Error: The process '/__w/1/s/gradlew' failed with exit code 1
##[debug]Processed: ##vso[task.issue type=error;]Code analysis failed. Gradle exit code: -1. Error: Error: The process '/__w/1/s/gradlew' failed with exit code 1
##[debug]Processed: ##vso[task.complete result=Failed;]Code analysis failed. Gradle exit code: -1. Error: Error: The process '/__w/1/s/gradlew' failed with exit code 1
##[debug]Handler exit code: 0
Async Command Start: Publish test results
##[debug]Uploading build level attachements individually
##[debug]Entering CreateTestRunAsync
##[debug]Leaving CreateTestRunAsync
##[debug]Total build level attachments: 0.
##[debug]Entering AddTestResultsAsync
Publishing test results to test run '********'.
TestResults To Publish 7, Test run id:***********
Test results publishing 7, remaining: 0. Test run id: **********
##[debug]Leaving AddTestResultsAsync
##[debug]Entering UpdateTestRunAsync
##[debug]Leaving UpdateTestRunAsync
##[debug]Entering CreateTestRunAttachmentsAsync
##[debug]Leaving CreateTestRunAttachmentsAsync
##[debug]Entering UploadTestRunLogAsync
##[debug]Blob name: GeneralAttachment/TestResults_***********.zip .
##[debug]File uploaded successsfully on LogStore TestResults_1359060.zip
Published Test Run : https://****************/*************/_TestManagement/Runs?runId=*********&_a=runCharts
##[debug]Leaving PublishAllTestResultsToSingleTestRunAsync
##[debug]Failed to get FF TestManagement.PTR.CalculateTestRunSummary Value.
##[debug]Failed to get FF TestManagement.Agent.PTR.EnableFlakyCheck Value.
Async Command End: Publish test results
Finishing: Execute gradle test

Repro steps

Run TestNG test suite containing multiple tests with Gradle.

Specify the path to TestNG generated JUnit reports for testResultsFiles input. Default path is build/reports/tests/test/junitreports where TestNG generates JUnit XML reports.

Enable TestNG listeners in build.gradle so it can produce reports:
test {
   useTestNG() {
     useDefaultListeners = true
   }
}

Here is how my pipeline job is configured:-

  - task: Gradle@3
    displayName: 'Execute gradle test'
    continueOnError: true
    inputs:
      gradleWrapperFile: 'gradlew'
      options: ${{ variables.gradleParameters }}
      tasks: test --info
      testResultsFiles: '**/build/reports/tests/test/junitreports/TEST-*.xml'

I am running my tests inside official gradle container:  

container:
    image: gradle:8-jdk17

The text was updated successfully, but these errors were encountered:

itkhanz · 2024-05-12T15:37:42Z

I have tried to resolve this issue by providing a path to merged merged junit xml report generated by org.testng.reporters.JUnitXMLReporter as:

  - task: Gradle@3
    displayName: 'Execute gradle test'
    continueOnError: true
    inputs:
      gradleWrapperFile: 'gradlew'
      options: ${{ variables.gradleParameters }}
      tasks: test --info
      testResultsFiles: "**/Gradle suite/Gradle test.xml" #path to merged JUNIT XML report generated by TESTNG

It resolves the issue of total run duration but causes another problem i.e. it does not ignore the tests that were not run and are marked as ignored in the XML. For example as you can see in the below merged xml output:

This test was not run during this build but test summary tab also includes this test as well as all others that were filtered out in the list of tests that were passed and assign zero duration to them which causes incorrect number of total tests.

Output should be like this in Test summary tab in Azure DevOps pipeline:

> Task :test FAILED

  4 passing (8m 36s)
  3 failing


---------------------------------------------------------------
|  Results: FAILURE (7 tests, 4 passed, 3 failed, 0 skipped)  |
---------------------------------------------------------------
Test report disabled, omitting generation of the HTML test report.

7 tests completed, 3 failed

but it shows all 68 tests (including those that were not run as tests that passed). Here is the debug output of Azure task:

##[debug]Object of JUnitResultParser created.
##[debug]Entering ParseTestResultFiles
##[debug]runContext.ReleaseURI is null
##[debug]runContext.ReleaseEnvironmentUri is null
##[debug]Reading test results from file '/agent/_work/2/s/build/testng/Gradle suite/Gradle test.xml'.
##[debug]Only single test suite found, parsing its information.
##[debug]Timestamp is not available for one or more testsuites. Total run duration is being calculated as the sum of time durations of detected testsuites.
##[debug]Total test results: 68.
##[debug]Leaving ParseTestResultFiles

The merged junit xml report looks like as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by org.testng.reporters.JUnitXMLReporter -->
<testsuite ignored="61" hostname="7f87e08ea7dc" failures="3" tests="7" name="Gradle test" time="513.435" errors="0" timestamp="2024-05-12T15:17:09 UTC">
  <testcase classname="com.dcs.appium.dcsapps.Tests.Favourites.FavouritesTest" name="VSDS_2406_VSDS_2408_Check_Adding_POI_As_Favourite_Then_Delete_It" time="3.361">
.
.
.
.
<testcase classname="com.dcs.appium.dcsapps.Tests.PoiDetailView.Reviews.Comment.VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button" name="VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button" time="0.0">
    <ignored/>
  </testcase> <!-- VSDS_7324_Check_Adding_Review_With_Comments_via_Review_Button -->

I see this could be the reason that Azure fails to recognize that attributes in testsuite tag which states total tests 7, failed 3, and ignored 61.

ivanduplenskikh · 2024-05-13T06:58:24Z

Hi @itkhanz thanks for reporting! We are working on more prioritized issues at the moment, but will get back to this one soon.

itkhanz added the bug label May 12, 2024

github-actions bot added Area: Release triage labels May 12, 2024

itkhanz changed the title ~~[BUG]: Total test run duration is incorrectly calculated~~ [BUG]: Total test run duration/count is incorrectly calculated from junit xml report May 12, 2024

kirill-ivlev removed the triage label May 13, 2024

v-schhabra added Area: ABTT Akvelon Build Tasks Team area of work and removed Area: Release labels May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: Total test run duration/count is incorrectly calculated from junit xml report #19857

[BUG]: Total test run duration/count is incorrectly calculated from junit xml report #19857

itkhanz commented May 12, 2024 •

edited

itkhanz commented May 12, 2024 •

edited

ivanduplenskikh commented May 13, 2024

[BUG]: Total test run duration/count is incorrectly calculated from junit xml report #19857

[BUG]: Total test run duration/count is incorrectly calculated from junit xml report #19857

Comments

itkhanz commented May 12, 2024 • edited

New issue checklist

Task name

Task version

Issue Description

Job Summary

Gradle logs

testng-results.xml

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server Version (if applicable)

Operation system

Relevant log output

Full task logs with system.debug enabled

Repro steps

itkhanz commented May 12, 2024 • edited

ivanduplenskikh commented May 13, 2024

itkhanz commented May 12, 2024 •

edited

itkhanz commented May 12, 2024 •

edited