Better handling of when no min MSI is provided & skip check in dry-run

[theofidry]

Depends on #1190

As of now there is two issues:

• If you provide a 0.0 min MSI score, you will not get suggestions to increase it if the score is above the suggestion threshold
• Same for min covered code MSI
• The MSI scores are checked even on dry-runs which does not makes sense since all mutations will be escaped in that mode

This PR addresses those 3 issues.

[sanmai]

The MSI scores are checked even on dry-runs which does not makes sense since all mutations will be escaped in that mode

There are plans to fix this issue by making them be caught in a pseudo-random fashion. As per #1150

[theofidry]

@sanmai do you think it's necessary? I feel like we can have an isolated scenario for profiling the reports rather than trying to set up some sort of fixtures in a dry run

[sanmai]

I do think this is necessary because leaving out reporting we get an incomplete picture. Say, it could happen a report keeps a handle to some larger object. If we can make a thorough dry run with all parts involved, we could notice this.

[theofidry]

But it's easier to detect that through a dedicated isolated scenario no? Plus we already have this sort of structure in our tests.

Also this allows to really test all reports in a consistent and similar fashion, whereas in a dry run, it depends on what report was configured

[sanmai]

Well, I'm not sure. Either way current approach just doesn't work for me. For example, you can easily trick a progress reporter to show current/max memory usage. But if there's no progress, as it is now, there's no reporting. Really inconvenient.

[theofidry]

@sanmai I would actually not include the progress reporter either, when profiling it's best combined with --quiet to avoid additional I/O. So in the same fashion adding more heavy I/O via reports is yet another part adding more instability to a profile. Whereas if you are interested in performances for a heavy report, it's gonna be trivial to adjust a scenario's fixtures

[sanmai]

Right. We have fine benchmarks for profiling purposes. If you can see #1150 isn't about profiling per se. Profiling isn't even mentioned there.

Shall I reopen #1150 then?

[theofidry]

If you are not using it for profiling, then putting different states is anything but confusing IMO. I would like however to make them as "skipped" later, which I also suggested in #1171

[sanmai]

"Skipped" won't do here because, say, if we were to change our logging procedure to be more efficient, any effect from the changes won't be seen.

[theofidry]

@sanmai I'm not sure to understand your concerns.

So far dry-run is only that, a dry run: you parse the files, create mutations, but don't process them.

Faking execution results to hit some "potential bottleneck parts" IMO fits more into a benchmark/profiling case or a "fake run" rather than a "dry-run". And if you don't execute the mutations at all, as suggested by the mode, it does not make sense to check the min MSI

[sanmai]

For example, we're collecting 100% of execution results, even if we can tell for certain we won't need some of them, or all of them, because of disabled logging. This is a definite area of improvement, but with the current state of --dry-run it is impossible to see any benefits from changes in the area.

[sanmai]

With all due respect, I don't think the part about dry run is a sensible change. It goes against the proposal from #1150.

[theofidry]

with the current state of --dry-run it is impossible to see any benefits from changes in the area.

But then again, we're back to profiling aren't we? And that's a part I would rather see profiled differently because very easy to do so without all the previous part noise

[sanmai]

Profiling with a tool like Blackfire is a final step. It is important, but not something I'd use as I code to quickly iterate over a feature or a change.