Aggregate coverage from calls to subprocess

[mojavelinux]

I have several CLI tests in my test suite. The lines covered by these tests are not being reflected in the coverage report. How do I get the coverage data collected from these calls to be incorporated into the coverage report? Is there an additional setting or environment variable that I need to set?

Here's the test suite: https://github.com/asciidoctor/asciidoctor-pdf

[marcandre]

Sorry for the delay. I can have a look.

IINM I need to run rake coverage test, right?

The first 3 uncovered spots I checked aren't indeed run, so which lines are you referring to?

[mojavelinux]

You need to run:

COVERAGE=deep rspec

Or you can just focus on the CLI tests:

COVERAGE=deep rspec spec/cli_spec.rb

(no need to use Rake)

[mojavelinux]

I'll provide a specific example for what I know is hit, but isn't covered. Stay tuned.

[mojavelinux]

Btw, I also frequently get core dumps when using deep cover, which is maybe associated with using a child process or maybe not related.

[mojavelinux]

Oh yeah, and I just remembered you can use the focused test because that leads to:

ree/util.rb:19:in `block in populate_from_map'
	 2: from /home/dallen/projects/asciidoctor/pdf/.bundle/gems/ruby/2.7.0/gems/deep-cover-core-0.7.10/lib/deep_cover/reporter/base.rb:39:in `block in populate_stats'
	 1: from /home/dallen/projects/asciidoctor/pdf/.bundle/gems/ruby/2.7.0/gems/deep-cover-core-0.7.10/lib/deep_cover/reporter/html/index.rb:14:in `block in stats_to_data'
/home/dallen/projects/asciidoctor/pdf/.bundle/gems/ruby/2.7.0/gems/deep-cover-core-0.7.10/lib/deep_cover/reporter/html/index.rb:52:in `transform_data': undefined method `transform_values' for "value_executed_loc_keys":String (NoMethodError)

^ It seems like deep cover fails in almost all cases when I try to filter the tests. I get all kinds of weird errors in the transform data routine.

[mojavelinux]

So here's a case. If I run the following command:

COVERAGE=deep bundle exec rspec -t cli

Then it says the truthy branch on line 222 (if use_title_page) is not covered. But if I write a debug statement to a file inside that condition, the file is written. So the code is clearly going there, but deep cover doesn't think so.

It appears some hit lines are recorded, but not all. It seems like there are logic errors in the merging of the coverage data.

[marcandre]

The reporting takes forever 😮
I have to check what's taking so long...

[mojavelinux]

The reporting takes forever

Tell me about it. But given how vital the information is, I've learned to be patient ;) If it could go faster, I would be 🙏.

[marcandre]

Ah, I think I found the issue; should be in the HTML reporter (and only the HTML reporter), and it makes sense too. I'm not sure yet of what approach I can take, I'll keep you updated.

[mojavelinux]

That's fantastic and encouraging to hear! Thanks for the update.

[marcandre]

I'm both quite ashamed and happy to report that, testing with some of your code (~500 loc), I get the reporting down from 56 seconds down to 0.8 seconds. Doubling the LOC quadruples the original time to a ridiculous ~4 minutes, while the optimized time is ~15% slower than twice the time at 1.9 seconds.

This is obtained by optimizing Parser::Source::TreeRewriter (no change to DeepCover per say). I need to tweak the code for older rubies (I used bsearch_index) and submit a PR to the Parser crew, but they are usually amazingly responsive.

Now I have to figure out what's going on with the actual coverage, but at least I don't have to wait minutes for the report...

[marcandre]

Created whitequark/parser#683 ...

[marcandre]

Updating parser should speed up the process a lot. I hope to look at the actual issue very soon...

[marcandre]

So I can confirm that the issue is that the code running via run_command isn't loading DeepCover. I put a raise 'hey, gotta load DeepCover' unless defined?(DeepCover) in converterd.rb and it fired when running the CLI tests. Our instructions weren't explicit about this, sorry, this so I've tried making it more precise.

In your case, since the code is loaded by asciidoctor, I'd recommend adding require 'deep-cover' if ENV['DEEP_COVER'] at the beginning of your main entry point lib/asciidoctor/pdf.rb and things should work fine and counters from both processes should be merged automatically. (you'll notice that you will have two .dct files in deep_cover instead of just one like currently).

Let us know if it doesn't work as it should.

[mojavelinux]

@marcandre Aha! That makes a lot of sense.

I'd recommend adding require 'deep-cover' if ENV['DEEP_COVER'] at the beginning of your main entry point lib/asciidoctor/pdf.rb

Eek! Testing code in production code. Not that. But I see what you mean. I'll use a supplemental script to load it.

Thanks for the pointers! I can't wait to see what the real numbers are.

[mojavelinux]

FYI, somehow nyc is able to do this in the Node world without the explicit require. I suspect they are doing some command manipulation. That's why I had just assumed deep cover would be loaded inside the CLI invocation.

[marcandre]

Interesting. I don't know how nyc does it. I believe that in the Ruby world it can't be done; simplecov for example needs something done (and not just a require).

[mojavelinux]

I'm pretty sure nyc intercepts the subprocess communication (though I'm fuzzy on the details at the moment).

[mojavelinux]

It seems to use NODE_OPTIONS to force itself to be required when the subprocess is called (assuming the subprocess is Node). This is similar to how Bundler bootstraps itself in a Ruby subprocess.