Json report #825

Bachmann1234 · 2019-07-15T05:01:17Z

The goal of this pr is to eventually lead to closing #720

Some example reports from a non trivial project
without Contexts: https://gist.github.com/Bachmann1234/418311192ebff1faca1e3fc54453602c
with Contexts: https://gist.github.com/Bachmann1234/bb799bdc5978a81a1e26daeb1881b950

command help:

venv ❯ coverage --help
Coverage.py, version 5.0a7 with C extension
Measure, collect, and report on code coverage in Python programs.

usage: coverage <command> [options] [args]

Commands:
    annotate    Annotate source files with execution information.
    combine     Combine a number of data files.
    erase       Erase previously collected coverage data.
    help        Get help on using coverage.py.
    html        Create an HTML report.
    json        Create a JSON report of coverage results.
    report      Report coverage stats on modules.
    run         Run a Python program and measure code execution.
    xml         Create an XML report of coverage results.

Use "coverage help <command>" for detailed help on any command.
Full documentation is at https://coverage.readthedocs.io/en/coverage-5.0a7

venv ❯ coverage json --help
Usage: coverage json [options] [modules]

Generate a JSON report of coverage results.

Options:
  --fail-under=MIN      Exit with a status of 2 if the total coverage is less
                        than MIN.
  -i, --ignore-errors   Ignore errors while reading source files.
  --include=PAT1,PAT2,...
                        Include only files whose paths match one of these
                        patterns. Accepts shell-style wildcards, which must be
                        quoted.
  --omit=PAT1,PAT2,...  Omit files whose paths match one of these patterns.
                        Accepts shell-style wildcards, which must be quoted.
  -o OUTFILE            Write the JSON report to this file. Defaults to
                        'coverage.json'
  --pretty-print        Print the json formatted for human readers
  --show-contexts       Show contexts for covered lines.
  --contexts=PAT1,PAT2,...
                        Only display data from lines covered in the given
                        contexts. Accepts shell-style wildcards, which must be
                        quoted.
  --debug=OPTS          Debug options, separated by commas. [env:
                        COVERAGE_DEBUG]
  -h, --help            Get help on this command.
  --rcfile=RCFILE       Specify configuration file. By default '.coveragerc',
                        'setup.cfg' and 'tox.ini' are tried. [env:
                        COVERAGE_RCFILE]

Full documentation is at https://coverage.readthedocs.io/en/coverage-5.0a7

simple report:

{
    "meta": {
        "version": "5.0a7",
        "timestamp": "2019-08-18T13:50:07.353011",
        "branch_coverage": true,
        "show_contexts": false
    },
    "files": {
        "a.py": {
            "executed_lines": [
                1,
                2
            ],
            "summary": {
                "covered_lines": 2,
                "num_statements": 3,
                "percent_covered": 60.0,
                "missing_lines": 1,
                "excluded_lines": 0,
                "num_branches": 2,
                "num_partial_branches": 1
            },
            "missing_lines": [
                3
            ],
            "excluded_lines": []
        }
    },
    "totals": {
        "covered_lines": 2,
        "num_statements": 3,
        "percent_covered": 60.0,
        "missing_lines": 1,
        "excluded_lines": 0,
        "num_branches": 2,
        "num_partial_branches": 1
    }
}

TODO:

```
Iterate on the schema
```
```
document the report in the formal docs
```

tests/test_json.py

+        cov.json_report(a, outfile=output_path)
+        with open(output_path) as result_file:
+            parsed_result = json.loads(result_file.read())
+        del(parsed_result['meta']['timestamp'])


nedbat · 2019-07-22T00:06:16Z

Thanks for this. I haven't had time to look at it yet, but will get to it soon!

Bachmann1234 · 2019-07-22T03:29:32Z

@nedbat no worries! I'm in no rush. I would love the feedback whenever you have time. However, I am not blocked as I still need to make time to finish up the tests and such.

codecov-io · 2019-08-01T02:50:47Z

Codecov Report

Merging #825 into master will increase coverage by 0.01%.
The diff coverage is 98.51%.

@@            Coverage Diff             @@
##           master     #825      +/-   ##
==========================================
+ Coverage   92.33%   92.34%   +0.01%     
==========================================
  Files          83       85       +2     
  Lines       11201    11302     +101     
  Branches     1141     1146       +5     
==========================================
+ Hits        10342    10437      +95     
- Misses        723      724       +1     
- Partials      136      141       +5

Impacted Files	Coverage Δ
coverage/config.py	`98.31% <100%> (+0.02%)`	⬆️
coverage/jsonreport.py	`100% <100%> (ø)`
coverage/xmlreport.py	`100% <100%> (ø)`	⬆️
tests/test_cmdline.py	`100% <100%> (ø)`	⬆️
tests/test_config.py	`100% <100%> (ø)`	⬆️
coverage/results.py	`96.53% <100%> (+0.02%)`	⬆️
coverage/control.py	`86.8% <100%> (-1.97%)`	⬇️
tests/test_json.py	`100% <100%> (ø)`
coverage/cmdline.py	`90.36% <80%> (+0.16%)`	⬆️
coverage/report.py	`98% <95.83%> (-2%)`	⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 790f0b3...d22c847. Read the comment docs.

Bachmann1234 · 2019-08-01T02:54:11Z

howdy! Just checking in to say im still alive. I updated my local master and then rebased this branch to keep it from getting stale. Im thinking ill have some time this weekend to beef up the tests a bit and look at the other todos.

Bachmann1234 · 2019-08-04T23:33:38Z

Ok, think I got decent tests on this the config file is tested so the only thing this needs (I think) is review/confirmation on the schema and then I need to write some docs.

Bachmann1234 · 2019-08-12T01:06:50Z

updated from master

nedbat

Matt, thanks so much for taking this on! My comments run the gamut, I hope you don't mind some pedantry on the English :)

nedbat · 2019-08-12T10:59:58Z

tests/test_json.py

+from coverage.misc import isolate_module
+from tests.coveragetest import UsingModulesMixin, CoverageTest
+
+os = isolate_module(os)


No need for this in our own test files.

nedbat · 2019-08-12T11:00:15Z

coverage/cmdline.py

+    )
+    json_pretty_print = optparse.make_option(
+        '', '--pretty-print', action='store_true',
+        help="print the json formatted for human readers",


nedbat · 2019-08-12T11:00:34Z

coverage/cmdline.py

+            Opts.contexts,
+            ] + GLOBAL_ARGS,
+        usage="[options] [modules]",
+        description="Generate an JSON report of coverage results."


nedbat · 2019-08-12T11:01:29Z

coverage/cmdline.py

@@ -755,6 +788,7 @@ def unglob_args(args):
            report      Report coverage stats on modules.
            run         Run a Python program and measure code execution.
            xml         Create an XML report of coverage results.
+            json        Create a JSON report of coverage results.


This should go between "html" and "report" (alphabetically)

nedbat · 2019-08-12T11:02:08Z

coverage/control.py

+        omit=None, include=None, contexts=None, pretty_print=None,
+        show_contexts=None
+    ):
+        """Generate an JSON report of coverage results.


nedbat · 2019-08-12T11:19:11Z

tests/test_json.py

+        output_path = os.path.join(self.temp_dir, "a.json")
+        cov.json_report(a, outfile=output_path)
+        with open(output_path) as result_file:
+            parsed_result = json.loads(result_file.read())


nedbat · 2019-08-12T11:24:51Z

coverage/jsonreport.py

+        self.data.set_query_contexts(self.config.report_contexts)
+        meta = {
+            "version": __version__,
+            "timestamp": str(int(time.time())),


Would ISO8601 be more friendly?

I think in my head I was thinking time.time() would be easier for machine parsing. However its not like ISO8601 would be hard... and it has the benefit of being much more human readable. So sure thing

Actually looking at this it may have been me doing what xml was doing. Changed to isoformat

nedbat · 2019-08-12T11:26:53Z

coverage/jsonreport.py

+            reported_file.update({
+                'arcs_missing': analysis.missing_branch_arcs(),
+                'arcs_executed': analysis.arcs_executed(),
+                'arcs_unpredicted': analysis.arcs_unpredicted(),


This value isn't usually in reports (I think?). I'd rather leave it out that confuse people at first.

Removed the references to arcs and just stuck with the branch percentages in totals

nedbat · 2019-08-12T11:27:53Z

coverage/jsonreport.py

+                'arcs_missing': analysis.missing_branch_arcs(),
+                'arcs_executed': analysis.arcs_executed(),
+                'arcs_unpredicted': analysis.arcs_unpredicted(),
+                'missing_branch_arcs': analysis.missing_branch_arcs(),


You've reported this twice (also as 'arcs_missing'). I wonder if we should avoid "arcs" in this report, since it's nerdy and internal. Reports tend to talk about "branches".

Also, this value is a dict, while "arcs_executed" is a list of pairs. Let's make them uniform, and both be a list of pairs.

nedbat · 2019-08-12T11:30:44Z

coverage/jsonreport.py

+            'missing_lines': nums.n_missing,
+        }
+        reported_file = {
+            'executed_lines': list(sorted(analysis.statements)),


analysis.statements is not the executed lines, it's the executable lines. And, awkwardly, it looks like Analysis computed "executed" but does not make it available...?

Analysis also has excluded, which could be in the report.

Bachmann1234 · 2019-08-12T12:40:13Z

Matt, thanks so much for taking this on! My comments run the gamut, I hope you don't mind some pedantry on the English :)

Not at all. If anything I should be a tad bit embarrassed for "an json" :-P

Looks like a mixture of small things and at least one or two big errors on my part to correct. I collected all the feedback. Left a few comments and will start working though this. Thanks!

Bachmann1234 · 2019-08-14T03:29:19Z

Handled a lot of the smaller feedback. Still have more to look at though

Bachmann1234 · 2019-08-18T18:00:11Z

#825 (comment)

This piece of feedback is still outstanding. We don't need the summary block but I find when looking at the full reports its nice to have these "summary stats" grouped together. Separating these properties from what I have perceived as the more raw data.

So I see it as a tradeoff between making the report a bit easier on the human eye while making it mildly more annoying for the computer to parse since you have to go down one more level to access these properties.

You see it when just scrolling though a non trivial report
https://gist.github.com/Bachmann1234/bb799bdc5978a81a1e26daeb1881b950

^^ diff cover with contexts enabled.

Bachmann1234 · 2019-08-18T18:00:38Z

I think all other feedback has been addressed and I updated the readme to reflect the updated state of the code

Bachmann1234 · 2019-08-18T18:05:06Z

updated pr from latest master

nedbat · 2019-08-24T21:18:09Z

tests/test_config.py

@@ -332,6 +332,10 @@ class ConfigFileTest(UsingModulesMixin, CoverageTest):
        hello = world
        ; comments still work.
        names = Jane/John/Jenny
+


Lint reports trailing space here. There's a .editorconfig file in this repo, if you enable it in your editor, it will handle these things automatically.

nedbat · 2019-08-24T21:28:19Z

On the structure of the JSON: OK, files will have a "summary" key.

This is very close to merging! You'll need to squash the commits. I can add docs, unless you want to take a pass at it. I wonder if people will want us to explain the JSON structure? I hope not.. :)

There's also CHANGES.rst and CONTRIBUTORS.txt, which I can tweak up if you like.

Bachmann1234 · 2019-08-25T03:42:08Z

added myself to contributors and added a line to changes. Squashed everything down to one commit. I can take a stab at the docs.

Bachmann1234 · 2019-08-29T01:33:10Z

doc/config.rst

+are outputted with whitespace formatted for human consumption (True) or for
+minimum file size (False)
+
+``json_show_contexts`` (boolean, default false): should the json report include


I stole the language for this line from the HTML section. I worry its a tad awkward because "on each line" is not quite right in the json context. Its not like the HTML report where every line literally has something I can hover over and see the contexts.

Its more like "Include data in the report showing the contexts recorded for each line"

Bachmann1234 · 2019-08-29T01:35:32Z

Ok @nedbat I took a stab at the docs. One place I considered making an update but left alone was the FAQ there is a question that references the XML and HTML reports

**Q: I use nose to run my tests, and its coverage plugin doesn't let me create
HTML or XML reports. What should I do?**

I could change that to 'HTML, JSON, or XML' reports if you want but I did not think it added much.

the doc changes are in this commit 436eff6

nedbat · 2019-08-29T23:58:38Z

@Bachmann1234 this looks great. I am ready to merge it, and will make some minor edits after that. Squash the commits, and let's get this to master-town! :)

Bachmann1234 · 2019-08-30T02:02:44Z

@nedbat Squashed down to one commit and ready to go

Bachmann1234 · 2019-08-30T02:05:03Z

I have a watch on the repo so ill keep an eye out for any issues around this (or if something else catches my eye). Im sure something will come up. Also you know how to get ahold of me :-)

Bachmann1234 commented Jul 15, 2019

View reviewed changes

Bachmann1234 force-pushed the json_report branch from f6e3564 to dd47d7e Compare August 1, 2019 02:50

Bachmann1234 force-pushed the json_report branch 2 times, most recently from f7020a4 to 52216fe Compare August 4, 2019 23:01

Bachmann1234 changed the title ~~[WIP] Json report~~ Json report Aug 4, 2019

Bachmann1234 force-pushed the json_report branch from 5757ce8 to ed97e33 Compare August 12, 2019 00:40

nedbat requested changes Aug 12, 2019

View reviewed changes

Bachmann1234 force-pushed the json_report branch from 3fa1fce to adc857a Compare August 14, 2019 03:27

Bachmann1234 force-pushed the json_report branch from 26ed1da to 22a1a64 Compare August 18, 2019 18:04

nedbat reviewed Aug 24, 2019

View reviewed changes

nedbat approved these changes Aug 24, 2019

View reviewed changes

Bachmann1234 force-pushed the json_report branch from c82c6da to 6740c2d Compare August 25, 2019 03:30

Bachmann1234 force-pushed the json_report branch 2 times, most recently from c60a8d3 to 436eff6 Compare August 29, 2019 01:29

Bachmann1234 commented Aug 29, 2019

View reviewed changes

Create a JSON report

d22c847

Bachmann1234 force-pushed the json_report branch from 436eff6 to d22c847 Compare August 30, 2019 02:02

nedbat merged commit 9a78a80 into nedbat:master Aug 31, 2019

nedbat mentioned this pull request Aug 31, 2019

Add a structured "report", probably JSON #720

Closed

Paebbels mentioned this pull request Jan 12, 2024

JSON coverage files should contain a format version #1732

Closed

Json report #825

Json report #825

Conversation

Bachmann1234 commented Jul 15, 2019 • edited

This comment was marked as resolved.

This comment was marked as resolved.

nedbat commented Jul 22, 2019

Bachmann1234 commented Jul 22, 2019

codecov-io commented Aug 1, 2019 • edited

Codecov Report

Bachmann1234 commented Aug 1, 2019

Bachmann1234 commented Aug 4, 2019

Bachmann1234 commented Aug 12, 2019

nedbat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bachmann1234 commented Aug 12, 2019

Bachmann1234 commented Aug 14, 2019

Bachmann1234 commented Aug 18, 2019

Bachmann1234 commented Aug 18, 2019

Bachmann1234 commented Aug 18, 2019 • edited

Choose a reason for hiding this comment

nedbat commented Aug 24, 2019

Bachmann1234 commented Aug 25, 2019

Choose a reason for hiding this comment

Bachmann1234 commented Aug 29, 2019 • edited

nedbat commented Aug 29, 2019

Bachmann1234 commented Aug 30, 2019

Bachmann1234 commented Aug 30, 2019

Bachmann1234 commented Jul 15, 2019 •

edited

codecov-io commented Aug 1, 2019 •

edited

Bachmann1234 commented Aug 18, 2019 •

edited

Bachmann1234 commented Aug 29, 2019 •

edited