Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add branch details to JSON report #1438

Closed
wants to merge 7 commits into from
Closed

Add branch details to JSON report #1438

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0bb7d56
Extend json report test
christoph-blessing Aug 18, 2022
b387359
Add branch details to json report
christoph-blessing Aug 18, 2022
99ed2db
Add exiting branch arc to json report test
christoph-blessing Aug 19, 2022
97b2ac8
Update branch details format
christoph-blessing Aug 19, 2022
d6f9893
Fix linting errors
christoph-blessing Aug 19, 2022
b8e1e50
Merge branch 'master' into add_branch_details_to_jsonreport
nedbat Sep 20, 2022
f0aa16f
Merge branch 'master' into add_branch_details_to_jsonreport
nedbat Sep 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
18 changes: 18 additions & 0 deletions coverage/jsonreport.py
View file
Open in desktop
Expand Up @@ -102,4 +102,22 @@ def report_one_file(self, coverage_data, analysis):
'covered_branches': nums.n_executed_branches,
'missing_branches': nums.n_missing_branches,
})
reported_file['branch_details'] = _report_branch_details(analysis)
return reported_file


def _report_branch_details(analysis):
"""Extract branch details for a single file."""
executed_arcs = analysis.executed_branch_arcs()
missing_arcs = analysis.missing_branch_arcs()
details = {}
for source_lineno in executed_arcs.keys() | missing_arcs.keys():
target_linenos_executed = executed_arcs.get(source_lineno, [])
target_linenos_missing = missing_arcs.get(source_lineno, [])
n_target_linenos = len(target_linenos_executed) + len(target_linenos_missing)
details[source_lineno] = {
'executed_branches': target_linenos_executed,
'missing_branches': target_linenos_missing,
'percent_covered': len(target_linenos_executed) / n_target_linenos * 100,
}
return details
88 changes: 55 additions & 33 deletions tests/test_json.py
View file
Open in desktop
Expand Up @@ -21,6 +21,10 @@ def _assert_expected_json_report(self, cov, expected_result):
a = {'b': 1}
if a.get('a'):
b = 1
elif a.get('b'):
b = 2
else:
b = 3
""")
a = self.start_import_stop(cov, "a")
output_path = os.path.join(self.temp_dir, "a.json")
Expand All @@ -43,34 +47,46 @@ def test_branch_coverage(self):
},
'files': {
'a.py': {
'executed_lines': [1, 2],
'missing_lines': [3],
'executed_lines': [1, 2, 4, 5],
'missing_lines': [3, 7],
'excluded_lines': [],
'branch_details': {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mm, in Slipcover, I chose to output executed_branches and missing_branches directly next to executed_lines, etc., as a list of tuples (which in JSON becomes a list of 2-element lists). With ->exit branches being shown as [x,0] "tuples" (where x is the source line).

We don't have to have the same format, but it would be nice...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks for the feedback. I changed the format to the one you requested.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Thank you.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious why 0 is better than -1 for exit?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mm, I think both work well. I just a hope Ed to pick 0. Do you have a reason why -1 is better?

.. Juan

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Internally, entering and exiting a code object is represented in arcs as negative numbers. From one of the docstrings:

        Negative numbers have special meaning.  If the starting line number is
        -N, it represents an entry to the code object that starts at line N.
        If the ending ling number is -N, it's an exit from the code object that
        starts at line N.

Here's some actual data from the .coverage file:

.coverage> select * from arc;
 +---------+------------+--------+------+
 | file_id | context_id | fromno | tono |
 +---------+------------+--------+------+
 | 1       | 1          | 3      | 4    |
 | 1       | 1          | 5      | -1   |
 | 1       | 1          | 12     | 13   |
 | 1       | 1          | 17     | 18   |
 | 1       | 1          | 9      | 17   |
 | 1       | 1          | 1      | 9    |
 | 1       | 1          | 18     | -1   |
 | 1       | 1          | 15     | 11   |
 | 1       | 1          | -1     | 1    |
 | 1       | 1          | 4      | 5    |
 | 1       | 1          | 12     | 15   |
 | 1       | 1          | 10     | 11   |
 | 1       | 1          | -9     | 10   |
 | 1       | 1          | 7      | 3    |
 | 1       | 1          | 4      | 7    |
 | 1       | 1          | 13     | -9   |
 | 1       | 1          | 2      | 3    |
 | 1       | 1          | 11     | 12   |
 | 1       | 1          | -1     | 2    |
 +---------+------------+--------+------+

I'm curious why you haven't seen negative numbers other than -1 in testing.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's easy to get other negative numbers. This is using the code from this pull request:

# /tmp/b1438.py

def never(x):
    print("ok")
    if x: return 17
    print("nope")

never(0)

then:

% coverage run --branch /tmp/b1438.py
ok
nope

% coverage report -m
Name                    Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------
/private/tmp/b1438.py       5      0      2      1    86%   5->exit
-------------------------------------------------------------------
TOTAL                       5      0      2      1    86%

% coverage json
Wrote JSON report to coverage.json

% jq <coverage.json
{
  "meta": {
    "version": "6.4.5a0",
    "timestamp": "2022-09-05T07:25:02.254430",
    "branch_coverage": true,
    "show_contexts": false
  },
  "files": {
    "/private/tmp/b1438.py": {
      "executed_lines": [
        3,
        4,
        5,
        6,
        8
      ],
      "summary": {
        "covered_lines": 5,
        "num_statements": 5,
        "percent_covered": 85.71428571428571,
        "percent_covered_display": "86",
        "missing_lines": 0,
        "excluded_lines": 0,
        "num_branches": 2,
        "num_partial_branches": 1,
        "covered_branches": 1,
        "missing_branches": 1
      },
      "missing_lines": [],
      "excluded_lines": [],
      "executed_branches": [
        [
          5,
          6
        ]
      ],
      "missing_branches": [
        [
          5,
          -3
        ]
      ]
    }
  },
  "totals": {
    "covered_lines": 5,
    "num_statements": 5,
    "percent_covered": 85.71428571428571,
    "percent_covered_display": "86",
    "missing_lines": 0,
    "excluded_lines": 0,
    "num_branches": 2,
    "num_partial_branches": 1,
    "covered_branches": 1,
    "missing_branches": 1
  }
}

missing_branches mentions -3, because the exit was never taken from the code starting at line 3.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For that .py, Slipcover outputs a missing (5,5) branch, since really it is a same-line branch (to the return statement), below. For branches that exit a function, it outputs 0 as the destination... As in the (2,0) branch for

def foo(x):
    if x == 0:
        x += 1
foo(0)

I could change that to -1, but to change that to something dependent upon where the code starts would take more effort.

{
    "meta": {
        "software": "slipcover",
        "version": "0.2.0",
        "timestamp": "2022-09-05T14:24:30.974682",
        "branch_coverage": true
    },
    "files": {
        "b1438.py": {
            "executed_lines": [
                3,
                4,
                5,
                6,
                8
            ],
            "missing_lines": [],
            "summary": {
                "covered_lines": 5,
                "missing_lines": 0,
                "covered_branches": 1,
                "missing_branches": 1,
                "percent_covered": 85.71428571428571
            },
            "executed_branches": [
                [
                    5,
                    6
                ]
            ],
            "missing_branches": [
                [
                    5,
                    5
                ]
            ]
        }
    },
    "summary": {
        "covered_lines": 5,
        "missing_lines": 0,
        "covered_branches": 1,
        "missing_branches": 1,
        "percent_covered": 85.71428571428571
    }
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need to talk about the value of coverage.py and slipcover producing the same data, vs coverage.py producing data with more information?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right... Is there truly value in encoding the starting line of the code in the branch destination? How would that help a user?

I guess the more the packages differ in the data they collect, potentially the more difficult a Slipcover-based collector for coverage.py might become... which, of course, at this point is just an idea.

In this case in particular, it seems like we'd generate different data anyway, because of the same-line branch.

'2': {
'executed_branches': [4],
'missing_branches': [3],
'percent_covered': 50.0,
},
'4': {
'executed_branches': [5],
'missing_branches': [7],
'percent_covered': 50.0,
},
},
'summary': {
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'num_branches': 2,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'num_branches': 4,
'excluded_lines': 0,
'num_partial_branches': 1,
'covered_branches': 1,
'missing_branches': 1,
'num_partial_branches': 2,
'covered_branches': 2,
'missing_branches': 2,
'percent_covered': 60.0,
'percent_covered_display': '60',
},
},
},
'totals': {
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'num_branches': 2,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'num_branches': 4,
'excluded_lines': 0,
'num_partial_branches': 1,
'num_partial_branches': 2,
'percent_covered': 60.0,
'percent_covered_display': '60',
'covered_branches': 1,
'missing_branches': 1,
'covered_branches': 2,
'missing_branches': 2,
},
}
self._assert_expected_json_report(cov, expected_result)
Expand All @@ -85,24 +101,24 @@ def test_simple_line_coverage(self):
},
'files': {
'a.py': {
'executed_lines': [1, 2],
'missing_lines': [3],
'executed_lines': [1, 2, 4 ,5],
'missing_lines': [3, 7],
'excluded_lines': [],
'summary': {
'excluded_lines': 0,
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'percent_covered': 66.66666666666667,
'percent_covered_display': '67',
},
},
},
'totals': {
'excluded_lines': 0,
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'percent_covered': 66.66666666666667,
'percent_covered_display': '67',
},
Expand Down Expand Up @@ -130,32 +146,38 @@ def run_context_test(self, relative_files):
},
'files': {
'a.py': {
'executed_lines': [1, 2],
'missing_lines': [3],
'executed_lines': [1, 2, 4, 5],
'missing_lines': [3, 7],
'excluded_lines': [],
"contexts": {
"1": [
"cool_test"
],
"2": [
"cool_test"
]
],
"4": [
"cool_test"
],
"5": [
"cool_test"
],
},
'summary': {
'excluded_lines': 0,
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'percent_covered': 66.66666666666667,
'percent_covered_display': '66.67',
},
},
},
'totals': {
'excluded_lines': 0,
'missing_lines': 1,
'covered_lines': 2,
'num_statements': 3,
'missing_lines': 2,
'covered_lines': 4,
'num_statements': 6,
'percent_covered': 66.66666666666667,
'percent_covered_display': '66.67',
},
Expand Down