Possible Incompatibility with SimpleCov 0.18+

[PragTob]

Hi there,

SimpleCov maintainer here.

To the best of my knowledge you're using resultset.json to do the code coverage reporting to code climate. Due to the introduction of branch coverage the format of this file will change slightly in the next release (beta release just released) - namely there are now top level "lines" and "branches" keys.

However, the file format might be changed further in the future due to problems with the standard branch format (we might even switch to YML format or something else). See simplecov-ruby/simplecov#801 for some information.

I don't want to break code climate test reporting, however:

a.) we need these changes to work correctly
b.) resultset.json was never intended or communicated as an integration point - it's merely the way in which we do merging results from different processes

As a result of this, I can't guarantee stability for resultset.json, furthermore as it pretty much just dumps and merges the standard coverage data it also doesn't support a lot of features of simplecov (off the top of my hat):

• ignoring coverage through nocov :nocov: is ignored with simplecov #246
• filtering out/ignoring files (all the different ways in which simplecov allows you to do this)
• tracking files/adding files that weren't loaded
• I'm not 100% sure if we write the file again after the last process finished writing, might be related to SimpleCov merged reports are not processed correctly #385

My suggested approach would be to write a gem/formatter that writes a file in a format that you control and can continue working. (I know this is against the general "no integration needed" idea of this project) I'm happy to work with you to ensure that the APIs you need are stable. I also still have hopes (:crossed_fingers:) to release a 1.0 version soon-ish (couple of months) and to make another pass through the code to make sure that public vs. private APIs in the code are accurately documented. Happy to provide you with more public APIs if you need them.

Cheers and happy new year,
Tobi

[efueger]

Thanks for the heads up @PragTob ! Pulling in a dev colleague to take a look.

[ale7714]

Hi @PragTob thank you for reaching out. We would really like to keep this project without requiring integration. Have you considered for simplecov to support a json output of the results? Other test coverage tools offer different formats for the outputs of the result. We would be happy on helping with writing that.

[PragTob]

Hi @ale7714 !

That's what I thought (although I think it's an integration either way) - you mean without adjusting the user simplecov code in any way, correct?

We have different output formats, supported through formatters (https://github.com/colszowka/simplecov/blob/master/doc/alternate-formatters.md, one of them is even a JSON formatter but it looks like it's unmaintained as of now: https://github.com/vicentllongo/simplecov-json ) :) Having a JSON output for simplecov is an idea, but if I understand your application correctly you'd rather that the user doesn't have to set anything - in other words you'd want the JSON report to be written on every run - correct?
Do you have an example of a JSON output from another coverage tool that you like?

I see how a JSON report could be useful, I just somewhat worry about the format of it (and then the complexity for you to parse it). Ruby now has 4 coverage criteria (lines, branch, method, oneshot lines) and the vision for SimpleCov is to allow all of them to be run individually or together. I guess a structure like:

{"file_name": {"coverage_method": data}}

could work. But one of the problems for your current way of integrating is that this would mean that you also need to be able to process all n types of coverage data we support.

Cheers,
Tobi

[ale7714]

Hi @PragTob,

in other words you'd want the JSON report to be written on every run - correct?
Yes that's correct.

XCov does something like this and it works out great. Handling the parsing on the test-reporter is fairly straightforward once we know how the template looks like. We have some tools that do lines and branch coverage and we're able to successfully parse the information we need once we get know how the format looks like.

I'll reiterate that we're happy to contribute to get the implementation of the JSON output up and running. Although I would defer to you on the template of the format. In general lines the information, currently provided in the resultset is enough for us but if we were able to add additional information link no-cov that would be awesome. Once we have a new template, on our side we will put together a new parser for it.

[PragTob]

Hi @ale7714,

it's nice that you offer help for the JSON formatter and I recognize that. I'd have to think up a suitable format and what to do, right now that's not my focus though.

The thing that irks me a bit here though is that in the end that we'd create and write a JSON report on every run while that's only necessary for the minority of runs that run on CI for a code climate customer. I made a quick check with 3 other online tools that provide code coverage and all of them integrate with us through a ruby gem (which imo is the right way to do it).
Feels odd right now that we'd have to do something to make the integration with Code Climate working not the other way around.

Does Code Climate still have an environment variable like "CODE_CLIMATE_TOKEN" or something that could be checked to only write the JSON report (unless configured in other ways) when that is set?

[ale7714]

Hi @PragTob,

Thank you for your response. I appreciate you’re taking the time to discuss this with us. I understand that other tools might do things differently. A couple of years ago, we took the direction of building a test coverage reporter tool that could parse output from different coverage tools with the main goal of having a consistent way to setup coverage reporting with Code Climate across different languages.

Before writing the tool, we performed some research and we were able to identify that most coverage tools offer a way to output a file (in JSON, XML, or similar) with the test coverage data that we would be able to parse.

Does Code Climate still have an environment variable like "CODE_CLIMATE_TOKEN" or something that could be checked to only write the JSON report (unless configured in other ways) when that is set?

Yes, users are setting CC_TEST_REPORTER_ID on their environment.

We would love to setup a time to chat with you to discuss how can we partner and support you on this. Please, let us know your availability over email at hello@codeclimate.com or a DM us in Twitter. Looking forward to hearing back from you.

[PragTob]

Thanks I'll get in touch although I thought communication here was mostly fine.

And forgot to update here, the version with the incompatible file format was released 2 days ago so this is now a current issue.

[maximeg]

Hi there,

Simplecov 0.18.0 is released and we could not use it as it leads to an error with cc-test-reporter:

#!/bin/bash -eo pipefail
./cc-test-reporter format-coverage -t simplecov -o coverage/codeclimate.json
./cc-test-reporter upload-coverage -i coverage/codeclimate.json

Error: json: cannot unmarshal object into Go struct field input.coverage of type []formatters.NullInt
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov, xccov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/circleci/project")

Global Flags:
  -d, --debug   run in debug mode

Exited with code exit status 255

[efueger]

Hi @maximeg - Emily here with Code Climate Support. Thanks for the report! We're working on this now.

• for now, the workaround is to use SimpleCov 0.17.x

If you have any other questions in the meantime, feel free to reach out to hello@codeclimate.com and we'll be happy to help.

[bestwebua]

@maximeg I have the same result. Sadly, but right now possible to use simplecov <= 0.17.x with latest cc-test-reporter

simplecov-ruby/simplecov#851 (comment)

[sgnn7]

Just a note that the main change is the introduction of 'lines' in the hierarchy of the output data from what I found so it may be an easy bandaid for now. Sadly, I don't have the logs present from when I was looking into this but the rest of the data looked the same. :(

Edit: I missed that the original post already mentioned this (oops!)

[PragTob]

@sgnn7 that is correct. See my initial post. The problem though is that the current "integration" also already suffers from multiple problems (such as ignored lines not working) and that the file used is internal and it's format might change soon - even drastically (it's an option that it becomes a YAML due to problems encountered with the branch format in JSON).

Sure CodeClimate could patch it up, but it's not the most long term stable solution. We'll see what we can up with, if anything.

[deivid-rodriguez]

I added bare support for simplecov 0.18 in #420 in case someone wants to give it a try.

[david942j]

I can see there are numerous PRs that keep using simplecov < 0.18 as the "solution" for this issue. Do we have a better solution here? I see #413 (comment) that removed coverageLocation from the config yml but for my project it never has that line.

[bmulholland]

From my experience, CodeClimate basically stopped development on their Quality product a couple years ago. Last I heard from them, about a year ago, I had sent in an email basically saying "What's going on? Everything's broken. Will it be fixed?" and they never got back to me, even with followups. So I wouldn't count on this ever being fixed.

[coding-bunny]

Can only add my voice to @bmulholland; We canceled our subscription with CodeClimate because of this issue and build our own code analyzer and tooling to run the things like RuboCop, Code Coverage etc.

[fede-moya]

Hey @david942j @bmulholland @coding-bunny This issue has been solved months ago. The only simplecov versions we don't support are 0.18.x and 0.19.0. We offer support for Simplecov 0.17 and earlier and for Simplecov 0.20+. For simplecov >= 0.19 the only thing that's needed is for the CC_TEST_REPORTER_ID env variable to be present at the moment simplecov is being run. If for some reason you don't manage to make it work you can email me directly at fmoya@codeclimate.com and I will help you resolve it.

[david942j]

Hi @fede-moya, it didn't work for my project when upgrading SimpleCov from 0.17 to 0.21: https://github.com/david942j/one_gadget/runs/5685500299?check_suite_focus=true

[david942j]

the only thing that's needed is for the CC_TEST_REPORTER_ID env variable to be present at the moment simplecov is being run.

Oh so you mean I need to set that variable for both bundle exec rake and the codeclimate action?

[fede-moya]

@david942j yeap

[david942j]

Got no luck :( https://github.com/david942j/one_gadget/runs/5695110552?check_suite_focus=true

[david942j]

For my project this issue got fixed by this PR: david942j/one_gadget#191

1. You need to set env CC_TEST_REPORTER_ID for both simplecov (i.e. bundle exec rake) AND the codeclimate action.
2. Explicitly setting SimpleCov.formatter would break things if you don't include the JSON formatter in your formatters. Explicitly setting the formatter as JSONFormatter should also work, and by doing so you don't need (1). Ref: https://github.com/simplecov-ruby/simplecov/blob/441d8ca6249c07275202880f3ff604272a4a3b76/lib/simplecov/default_formatter.rb#L11

[pboling]

I wrote a new gem that uses the latest gem "simplecov", "~> 0.21", successfully with CodeClimate... and CodeCov, and Coveralls, and CodeCoverageSummary.

Here's how I did it!

• Example Github Actions Build
• Code Coverage Workflow
• setup-code-climate Github Action I use
• Coverage-related Gems I use
• require "simplecov" before loading the code to be coverage checked

There is no longer a compatibility issue.

I wrote a more complete article about the setup on dev.to

[jakeonfire]

@pboling simplecov now includes a JSON formatter, imported with:

require "simplecov_json_formatter"