Regression: exit code 0 when loaded after minitest

[searls]

This seems like a regression of #269

In a very simple project if I have this Rakefile:

require 'rake/testtask'

Rake::TestTask.new do |t|
  t.libs.push 'test'
  t.pattern = 'test/**/test_*.rb'
end

task :default => :test

And this test helper:

require "minitest/autorun"

require 'simplecov'
SimpleCov.start

And an error occurs, the process exits code 0. However, if I correct my configuration to load simplecov before minitest/autorun:

require 'simplecov'
SimpleCov.start

require "minitest/autorun"

exits non-zero.

Obviously, I had simplecov misconfigured, but I got no feedback at all I was doing it wrong, and my build was passing erroneously. Is there anything simplecov can do to guard against this or at least warn me?

[bf4]

That's because minitest/autorun is kind of insane and doesn't start until the program ends. Yes, that's right, minitest autorun runs your entire test suite in an at_exit block. So, that's why you needed to require simplecov first. Otherwise, minitest will finish after simplecov and your final exit code will be that of minitest. 😡

Requiring simplecov first loads the (default) at_exit hooks. I believe you can start simplecov after minitest as long as you have required simplecov first, but would want to confirm.

I ran into a lot of trouble with this in rails-api/active_model_serializers#1386 (comment) and rails-api/active_model_serializers#1109 which referencs http://blog.arkency.com/2013/06/are-we-abusing-at-exit/

[bf4]

So, I don't think it's a regression as that issue had to do with simplecov mishandling the error code. See my comment here on how I confirmed that #281 (comment)

If you can confirm, this would make a good PR to the docs.

[searls]

Wow, @bf4, thanks for the background. I won't try speculating about who/what/where and ways to improve things here, but I'll make a note of it.

[bf4]

Thanks. As an aside, I think we still wanted to get the rake task to eager
load, which could help

On Mon, Feb 22, 2016 at 7:55 AM Justin Searls notifications@github.com
wrote:

Wow, @bf4 https://github.com/bf4, thanks for the background. I won't
try speculating about who/what/where and ways to improve things here, but
I'll make a note of it.

—
Reply to this email directly or view it on GitHub
#465 (comment)
.

[PragTob]

Another thing that I hope #756 might solve