Make logging skipped scenarios optional

[zettpunkt79]

Adresses: Testrunner is flooded with skipped scenarios

Given I have a lot of scenarios in my feature file
And I add a focus tag to one scenario
When I run the feature in the test runner
Then I only want to see the execution log of the scenario with the focus tag
And I don't want to wait for the script runner to mark all other scenarios as skipped

Adresses: #419

Given I set the environment variable logSkipped to false
When I have some tests skipped during test execution
Then the cucumber report files do not include skipped scenarios
And I can successfully upload my report files to Zephyr Scale formerly known as TM4J

[badeball]

Can you elaborate on this, please? Am I to understand that you do in fact have a bunch of skipped scenarioes, but you do not want them logged as such?

[zettpunkt79]

Can you elaborate on this, please? Am I to understand that you do in fact have a bunch of skipped scenarioes, but you do not want them logged as such?

Yes. When I have written down my scenarios for a feature I usually start by annotating a scenario with a focus tag, open the cypress test runner and start developing my test. This worked great until #288 was merged. Now the test runner lists all the skipped scenarios too (which takes additional time) and the results of the test I'm focussing on get's scrolled out of view, so I spend a lot of time scrolling.

Second problem I have is that Zephyr Scale, which is a test management plugin for Jira, currently cannot handle scenarios with the status skipped, so I want to keep them out of the *.cucumber report files to be able to upload my test results.

Does this clear it up?

[badeball]

Does this clear it up?

No. With regards to the above-mentioned PR, is the JSON reporting-format well-defined somewhere and are we breaking it?

[zettpunkt79]

No. With regards to the above-mentioned PR, is the JSON reporting-format well-defined somewhere and are we breaking it?

I'll put it this way: v2.3.1 produced a JSON report that our test management plugin can handle just fine. All following versions produce a JSON report which includes skipped scenarios by default and our test management plugin cannot handle these, although the format of the report is well-defined. As I am not the only one who ran into this issue, see #419, I created this PR which keeps the new default behavior of reporting skipped scenarios but gives those, who don't want that, a chance to fall back to the behavior of v2.3.1 by using an env variable.

[badeball]

This is a duplicate of #474.

Let me make it abundantly clear - I have zero desire to maintain additional, compatibility modes of the JSON-format. The official formatter from Cucumber is going away and I'd rather make it go away entirely here as well. If some unmaintained tool can't handle it, then you should write your own post-processor that removes them.

With that said, I also greatly dislike #329, because it ensures that hooks run before skipped tests, which is an absolutely waste of time.

The most reasonable change in my eyes, is to not invoke state.onStartScenario and state.onEndScenario inside an it-block, but rather just do it synchronously and maybe run something along it.skip(scenario.name). That however is going to ensure that skipped tests still appear in the report, which is not what you want.

[zettpunkt79]

Closed in favor for #474