Also include test suite stdout/stderr in annotation details

[sfc-gh-zpeng]

#358 added stdout/stderr of testcases which is very nice. However, tools like bazel has the capability to generate a JUnit XML in which the stdout of the test program is attached as an attribute under testsuite. This is especially true in the following cases:

• the test is a general program whose structure bazel does not understand -- it treats the program as a testsuite
• the test is a python test, but it failed to run (like there's an import error, etc -- the test program died before the test infrastructure was able to run any test).

An example of such a JUnit XML output is like the following:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="bazel/failing_absl_test" tests="1" failures="0" errors="1">
    <testcase name="bazel/failing_absl_test" status="run" duration="0" time="0"><error message="exited with error code 1"></error></testcase>
      <system-out>
Generated test.log (if the file is not UTF-8, then this may be unreadable):
<![CDATA[exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //bazel:failing_absl_test
-----------------------------------------------------------------------------
Traceback (most recent call last):
  File "<reducted>", line 3, in <module>
    import non_existent_package
ModuleNotFoundError: No module named 'non_existent_package']]>
      </system-out>
    </testsuite>
</testsuites>

Does it make sense to also attach the system-out of testsuite to the annotation details?

[EnricoMi]

Yes, it makes sense. Thanks for the example XML.

I am wondering how that test suite output should be reported. Should it be added to all test-case annotations of the respective suite?

[sfc-gh-zpeng]

I guess so. Bazel's output (the example) is just a special case where there is one test case. There could be no test case (only a test suite) in some cases (if that's valid)?

[EnricoMi]

Since it is difficult to reason about which test case to associate the test suite stdout/stderr, I'd say an individual annotation would work best. Stdout could be info, stderr could be warning. But that would mean there is one or two annotations per testsuite, so this should be an opt-in.

[EnricoMi]

@sfc-gh-zpeng what do you think: #392