Coverage analysis stops before all tests have finished when using the gem minitest-optional_retry

[michel-zimmer]

When requiring the gem minitest-optional_retry the coverage analysis stops before all tests have finished.

Output from before adding the gem:

bin/rails test
Running via Spring preloader in process 16635
Run options: --seed 9134

# Running:

.....

Finished in 0.119671s, 41.7812 runs/s, 58.4937 assertions/s.
5 runs, 7 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to [...]/rails-simplecov-minitest-optional_retry-mcve/coverage. 25 / 29 LOC (86.21%) covered.

Output from after adding the gem:

bin/rails test
Running via Spring preloader in process 16735
Coverage report generated for MiniTest to [...]/rails-simplecov-minitest-optional_retry-mcve/coverage. 0 / 44 LOC (0.0%) covered.
Run options: --seed 56425

# Running:

.....

Finished in 0.111954s, 44.6613 runs/s, 62.5258 assertions/s.
5 runs, 7 assertions, 0 failures, 0 errors, 0 skips

Everything is on most recent release:

ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
gem 'rails', '~> 5.1.6'
gem 'minitest-optional_retry', '~> 0.0.2'
gem 'simplecov', '~> 0.16.1'

This file contains everything the gem actually does:

require 'minitest/autorun'

module Minitest
  module OptionalRetry
    def run_one_method(klass, method_name, reporter)
      report_result = nil
      3.times do
        result = Minitest.run_one_method(klass, method_name)
        report_result ||= result
        (report_result = result) and break if result.passed?
      end
      reporter.record(report_result)
    end
  end
end

(Taken from https://github.com/appfolio/minitest-optional_retry/blob/22490f82088cf13b2bf1abee8ebb3fc8050bd543/lib/minitest/optional_retry.rb)

I've create a mcve to be sure. It's source code can be found here (direct link to commits view). It consists of creating a Rails app, scaffolding something for testing, adding SimpleCov and then adding them gem.

[michel-zimmer]

@spjsschl, you might be interested in this.

[bf4]

Interesting. I'm almost certain this has to do with the fact that minitest runs in an at_exit hook, and SimpleCov does generates its reports in an at_exit hooks, and the order of the two.

It's also possible it's Spring related. I haven't looked too carefully at the https://github.com/michel-zimmer/rails-simplecov-minitest-optional_retry-mcve app yet. Thanks for that.

[PragTob]

Might potentially be fixed with #756

[PragTob]

@michel-zimmer do you want to re-test this? I hope it's fixed with #756 which I'll release soon but is already on master.

[michel-zimmer]

Hi @PragTob, thanks for your patience.
I've tested it again, using 0.18.5 for the mvce linked above, and the problem is still there.

Though I'm not developing with Ruby anymore and therefore don't depend on this problem being solved and also I won't be of much help either: I've got the mvce barely updated and running let alone debugging the problem properly ;)

Br, Michel

[PragTob]

Well thanks :) Might take a look but it's always hard to see in these things how it happens. As mentioned before, I suspect some at_exit weirdness.