[SUREFIRE-1934] Ability to disable system-out/system-err for successfuly passed tests. #670
Conversation
…uly passed tests. Added ability to disable system-out and system-err blocks.
…uly passed tests. Added it-tests.
| <artifactId>log4j</artifactId> | ||
| <version>1.2.16</version> | ||
| </dependency> | ||
| </dependencies> |
There was a problem hiding this comment.
Can you please swap this for SLF4J API + Simple. Log4J is a troublesome fellow.
| * <br> | ||
| * True by default. | ||
| * | ||
| * @since 3.0.0 |
| @@ -679,6 +679,16 @@ public abstract class AbstractSurefireMojo extends AbstractMojo implements Suref | |||
| @Parameter(property = "enableAssertions", defaultValue = "true") | |||
| private boolean enableAssertions; | |||
|
|
|||
| /** | |||
| * Flag for including/excluding system-out and system-err elements in xml reports. | |||
| /** | ||
| * Flag for including/excluding system-out and system-err elements in xml reports. | ||
| * <br> | ||
| * True by default. |
There was a problem hiding this comment.
These two lines are redundant, Plugin Tools will take care of that.
|
Also, were can I see that this applies to passed tests only? |
|
@NissMoony Please let me know whether you want to be merged. |
Yes, it seems that the @NissMoony, I guess all suggested changes are rather trivial to implement. If you give this PR some attention now, @michael-o, who is currently working on a bunch of at least two other output-stream-related PRs, could move forward more quickly and consider them all together as a whole. @michael-o, I am not a committer, but I guess you can just commit those trivial changes on top of the PR branch yourself, if you think the PR is valuable and otherwise OK. I have not reviewed it, just found it by chance. |
|
Without having seen the code, the basic idea sounds appealing. I lately had discussions in another project, because there were some tests with a lot of output, including some (expected) logged exception stack traces, which annoyed the reviewers, while I thought it was important to see the output in case of a test failure. So, to retain the output in case of an error and to suppress them if the tests pass, it actually a good idea. It can be dangerous too, if tests pass for the wrong reasons (e.g. weak assertions) without anyone noticing, but that is another topic. Over in the other project, I was urged to mock stdOut and stdErr to silence the tests, but that is not thread-safe, so I had to mark the tests as Update: OK, I quickly looked at the code. This PR does nothing of the sort I was talking about above. It does not change stdOut/stdErr capturing in any way, just adds a switch adding the output to the XML report. I.e., which ever way Surefire handles those streams in a multi-threading situation, remains the same. Sorry for the misunderstanding and for the noise. I had hoped that someone came up with a magical solution for the root problem. But it is not being addressed here. |
Yes, the problem really important fo our project. Sorry for long response, not much time in the end of year. I'll fix remarks, but now I can see one problem.
I'm little misunderstood issue that I linked with. For few our projects there is important to disable all output elements for all success and failure tests (how it worked in previous versions). |
I am not Michael, but IMO it makes sense to have both. The simplest way to do that would be two boolean options, one for failed tests and tests in error and one for passing tests. An alternative would be some kind of list with a set of fixed values, but then you also need to add all possible combinations to the list and name them accordingly. With two options, this is not much, but what if in the future we want to support more test statuses, e.g. differentiate failed tests from tests in error, or what if skipped tests in some tools can print the reason for skipping to the console? You get the picture. I am sure, @michael-o can guide you towards a good design decision. |
Scratch my previous statement. Re-reviewed your PR, from a static PoV, it looks reasonable. In fact, I picked up your idea and created #702. After it is merged, I will go over to your PR. |
| * @since 3.0.0 | ||
| */ | ||
| @Parameter(property = "enableOutputElements", defaultValue = "true") | ||
| private boolean enableOutputElements; |
There was a problem hiding this comment.
Please rename to enableOutErrElements this will better reflect the intent w/o even reading the docs.
|
I have now re-read the code and your change. I fail to see how this change only applies to successful tests. It applies to all type of results: |
|
Hey @michael-o, according to a previous comment it seems that the intention of @NissMoony was to provide an option for suppressing all output (irrespective of the type of the result) from the XML report. For our use-case (Apache Hive), I think that disabling all output from the XML report is sufficient to move our upgrade further assuming that Having more fine grained configurations (e.g., only apply this on successful/failed tests) may be a nice to have feature but not a strong requirement from our side at this point. We all understand that complex configurations are hard to maintain. |
The code contradicts the issue summary. Either one should be picked. |
Sure that is an obvious problem. It seems though that we all kind of agree with the idea in the code so we could just edit the summary and move this forward. |
I believe that it was wrong from begin with to make this non-optional. Means: Output for successful tests should be disabled by default and not the other way around. |
|
Personally, most of the time I would prefer not to have the STDOUT/STDERR in the XML report especially for tests that run successfully. I think the best would be to configure loggers accordingly and don't have to play with the surefire configuration but sometimes this is inevitable. |
|
I believe most of developers want to have logs for failed tests. Because often it's what we only have to investigate test failures on CI. Sometimes for rarely failing tests it's the only way to diagnose the problems. |
Added ability to disable system-out and system-err blocks through the plugin configuration.
For some tests the output too big and our pipeline tool cannot parse it.
Added it-tests.