Models in subfolders are reporting 0% coverage

[darkcode85]

Rails Version: 5.2.0.rc1
Simplecov Version: 0.15.1
Test Framework: Minitest

Test coverage are running properly for all the models in my application except for these that live in a sub folder, example:

I have a notification.rb model in app/models/notification.rb:

class Notification < ApplicationRecord
  belongs_to :user
end

And also a subclass model called email.rb in app/models/notification/email.rb:

class Notification::Email < Notification
   validates_presence_of :subject, :body
end

I have added the initial test coverage for both models. In test/models/notification_test.rb:

require 'test_helper'

class NotificationTest < ActiveSupport::TestCase
  should belong_to(:user)
end

and in test/models/notification/email_test.rb :

require 'test_helper'

class Notification::EmailTest < ActiveSupport::TestCase
  def setup
    @notification_email = build(:notification_email)
  end

  test 'should has a valid factory' do
    assert @notification_email.valid?
  end
end

Both tests are running and passing with success and I can force the test to fail but the coverage report always show 0% coverage for these models.

This is my test_helper.rb:

require 'simplecov'
SimpleCov.start :rails do
  add_filter 'app/models/application_record.rb'
  add_filter 'app/channels/application_cable/channel.rb'
  add_filter 'app/channels/application_cable/connection.rb'
end

ENV['RAILS_ENV'] ||= 'test'
require_relative '../config/environment'
require 'rails/test_help'

class ActiveSupport::TestCase
  include FactoryBot::Syntax::Methods
end

[PragTob]

👋

Thanks for the bug report!

That's super weird... Can you check if the problem really goes away if you move them outside of the subfolder? Or is it likely messing up autoloading.

I'm not sure if it's a problem with the folder, it might be the subclassing or it might be the autoloading of subfolders in rails.

It might also be a problem with the files being loaded before you run the tests (often with spring, sometimes while running with rake) - so it's likely an environment problem I think. Because the ruby coverage library takes care of tracking all files loaded after the test was invoked.

Without a repo to reproduce it's hard to figure out.

[darkcode85]

@PragTob thanks to reply, my mate has created a very simple repository to demonstrate the issue, you can clone it: git clone https://github.com/darkcode85/simplecov-bug.git

The repository has three basic models blog.rb to demonstrate that simplecov is working andnotification.rb with a subclass email.rb that live in a subfolder as I already explained in the issue, all three models are tested and here you can see the coverage output:

Hope you can replicate the issue with the repository.

[PragTob]

👋

Thanks for the quick turnaround, I'll try to set it up now, but can't guarantee how far I'll get - time being limited and all :)

linke for my future self: https://github.com/darkcode85/simplecov-bug

Also, how do you usually run the tests?

[darkcode85]

@PragTob Thanks! we run the test using: rails test

[PragTob]

Yeah can reproduce

[PragTob]

Works when running rake test - it has to do with Rails and their binstubs or whatevers as expected.

This is the rails binstub:

#!/usr/bin/env ruby
begin
  load File.expand_path('../spring', __FILE__)
rescue LoadError => e
  raise unless e.message.include?('spring')
end
APP_PATH = File.expand_path('../config/application', __dir__)
require_relative '../config/boot'
require 'rails/commands'

It loads a bunch of stuff before executing the tests, and as a result the coverage library can't track what was loaded and so it's wrong.

Might need to talk to the rails team as to how improve the situation or might have to look at something else.

[darkcode85]

@PragTob thanks to look into that, I can confirm that it works running rake test although it raises a lot of warnings. Can we expect a fix here as now rails test is the standard? or this is more a rails bug?

[PragTob]

this is neither really a rails bug nor a simplecov bug - it's a setup bug. As Rails is bigger and we integrate with Rails it's probably more of a documentation bug on our side ;) To be fair, our document is quite vocal about the fact that simplecov needs to be loaded and started first for it to work which is clearly violated here :)

I guess I gotta ask the rails folks to see what they think how this should best be handled/where it should be integrated.

[PragTob]

I did another round of testing... removing my old adversary spring via spring binstub --remove --all solves the problem... :|

[darkcode85]

@PragTob weird... I can confirm that after do spring binstub --remove --all the coverage report a 100% in these models.

[PragTob]

yeah spring messes with code coverage greatly due to the nature of what it does... we have specific spring instructions in the README but these don't work for me either - I haven't used spring myself in years as it caused me too much pain and too many lost hours already

[AlexTatarnikov]

Experiencing the same issue for rspec.

[0xifis]

Yeah, @PragTob, had the same elusive issue with spring. Similarly, I have decided to remove spring for rspec spring binstub rspec --remove. Amount of time spent finding a fixing a bug >> time saved using spring.

That being said, while trying to recreate this issue again, I noticed another weird interaction with spring.

1. I moved the simplecov setup to the spring initializer, config/spring.rb, as suggested in the readme.
2. Similar to @darkcode85, only some of the code was accounted for in the coverage.
3. Then, stopping the spring server with spring stop, caused the coverage results to be updated with the correct coverage results.

I dont know how useful that is but I thought I should mention it. Perhaps, someone with deeper understanding of spring might be able to explain this behaviour.

[bf4]

Debugging SimpleCov file loading, the movie:

export DISABLE_SPRING=1
cat <<'EOF' > debug_coverage.rb
require 'pp'
require 'coverage'
Coverage.start
at_exit { pp(Coverage.result) }
Rake::Test['default'].invoke # or whatever is a way to run the above before any requires in your Rakefile or spec or test helper
EOF
bundle exec ruby debug_coverage.rb

also looking at #465 #340 #671 (comment) #235 #328 (comment) #281

[PragTob]

Yeah.. I left this open so far to reinvestigate how we can work with spring as I think last time I tried it I forgot something.

I fear we won't work well with spring at all and I don't have much personal experience because spring has caused me more hours of debugging than saving time so I remove it from all my projects immedeatly.

[andypearson]

Just chiming in to say, @saramoohead and I were just trying to report coverage on some new work she is doing and Simplecov was returning some very wacky numbers (0% on models that had a 100% covering spec). Issue went away when re-running with DISABLE_SPRING=true.

Annoyingly, we actually ran into this problem 6 months earlier. We ended up adding DISABLE_SPRING=true to a build script that we have for generating full codebase coverage across RSpec and Cucumber.

Whilst not ideal, I wonder if it is possible for Simplecov to know if it's running inside Spring, and if so, produce a warning that recommends re-running using DISABLE_SPRING=true.

[bf4]

@andypearson I also disable spring all the time. I have this in my bash profile.

I suppose you can try something like caller.grep(/spring/) and see if it's in there.

The reason this is a problem at all is because you can't restart the Ruby Coverage module. So, with spring, the code is loaded before coverage and so is never added.

[r4mbo7]

Similar issue with Rails 6, Minitest, Spring and simplecov 0.17.1.
I run my tests with rails test

[vanessasoutoc]

Use rake test.
For me working now.