Coverage regressions on 3.10.rc1

[skirpichev]

See up to date codecov status and the build artifact. Coveragepy at 57a691f.

I'll select some cases (codecov shows more regressions, but only ones in the power.py and rde.py seems to be in the build artifact, perhaps it's a codecov issue).

1. L1032, L1037 of power.py: no jumps to exit. These lines could be covered, for example, by: exp(Symbol('', real=True, transcendental=True)).is_algebraic and exp(Symbol('', integer=True)).is_algebraic.
2. L309 of rde.py and similar lines with the DecrementLevel context manager. These can be triggered by tests in the diofant/tests/integrals/test_rde.py.

PS: You can see more "green" lines counted in the codecov statistics. Maybe it's a codecov issue. I've reopened the pr request as diofant/diofant#1162 to see if this and mentioned above disagreement with the codecov.xml in the build artifact will disappear.

[nedbat]

Please provide specific instructions about how I can reproduce the problem. Specific commits of your repo to clone, commands to run, and specific lines that you believe are incorrectly measured.

[skirpichev]

I though I did this...

Ok, next try. You can use diofant/diofant#1162, last commit (c59bd75).

1. L1032 and L1037 of the diofant/core/power.py are not covered (no jumps to exit). This should be covered by tests:

exp(Symbol('', integer=True)).is_algebraic  # L1032
exp(Symbol('', real=True, transcendental=True)).is_algebraic  # L1037

In principle, pytest --cov diofant -m 'not slow' diofant/tests/core should also cover these lines.

2. L309 of the diofant/integrals/rde.py (and similar lines with the DecrementLevel context manager). I'm unsure, maybe it's not a bug, but this line was marked as covered on the master, see the codecov output. Now coveragepy report "line 309 didn't except from function 'bound_degree', because the raise on line 320 wasn't executed or the raise on line 342 wasn't executed" (pytest --cov diofant -m 'not slow' diofant/tests/integrals/test_rde.py should trigger all tests for the bound_degree() function).

[skirpichev]

FYI: here is the codecov bugreport

[nedbat]

I've reported the L1032 issue against CPython: https://bugs.python.org/issue44840

[nedbat]

For the L309 issue, I see this in my coverage results:

It has executed line 336, but then neither line 339 or line 341. That makes it seem like some other exception was raised at line 336, which could explain why the context manager on line 309 never jumped to 371. Can you confirm this theory?

[skirpichev]

Hmm, I did pytest -m 'not slow and not xfail and not regression' diofant/tests/integrals/. It seems, L344 (exit from with) has hit from the test_integration() of diofant/tests/integrals/test_integrals.py.

[nedbat]

@skirpichev I'm sorry, I don't understand what your last comment means.

I tried adding another except clause. It shows that line 336 is raising an uncaught exception:

So I think the L309 issue is not a regression.

[skirpichev]

I don't understand what your last comment means.

Well, perhaps I misunderstood you, but it seems your guess was that the only exit from the with clause was by an exception (from L336). That's not true. E.g. you can replace L344 with print(111) (i.e. right after "with" statement) and run pytest -q -n0 -s diofant/tests/integrals/test_integrals.py -k test_integration - you will see some hits. Or run in console from diofant import *; integrate(exp(-log(x)**2)).

So I think the L309 issue is not a regression.

Clearly, there is an exit by an uncaught exception AND a normal exit.

I'll try to reduce this example, of course.

[skirpichev]

PS: You can see more "green" lines counted in the codecov statistics. Maybe it's a codecov issue.

Regarding this, it seems to be a coveragepy "feature". Indeed, some new lines are counted. For instance, L1489 of rings.py, which is "while True" statement.

I'll try to reduce this example, of course.

Another example is in the cancel_primitive():

coverage.xml:
<line number="546" hits="1" branch="true" condition-coverage="66% (2/3)" missing-branches="exit"/>
IIUC, three branches are: normal exit from "with", an uncaught exception from some function inside of "with" and the NotImplementedError on L552?

Probably, you were right, this is not an issue. On another hand, in this example we have only a NotImplementedError, which is ignored by my config. Shouldn't then this branch be ignored by the coveragepy (like for if/else blocks)?

At least, the coveragepy behaviour was changed significantly, could be this documented?

[skirpichev]

FYI, unless I miss something, 1st issue wasn't solved in cpython:

Coveragepy at 57a691f. I've tested this issue on the recent 3.10 head, which has backport from your issue:

$ python
Python 3.10.0rc1+ (heads/3.10:d5c8ad2471, Aug  9 2021, 15:31:13) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

[nedbat]

@skirpichev I really appreciate your dedication to finding these problems, but please please please please please PLEASE always include the steps you used to produce the results you are showing.

And can you say specifically what about that output is incorrect?

This is what I did (with output showing versions):

$ mktmpenv -p /usr/local/cpython/bin/python3 -n
$ python -VV
Python 3.10.0rc1+ (heads/3.10:d5c8ad2471, Aug  9 2021, 10:24:02) [Clang 12.0.0 (clang-1200.0.32.29)]
$ cd diofant
$ git log --oneline | head -1
c59bd753e XXX removed all workarounds for issue2506 & skipif ground types != gmpy
$ pip install -e '.[develop,docs,tests]'
$ pip install git+https://github.com/nedbat/coveragepy@57a691f15dc8296f91ec1e321d3ed53e165a1f10
$ pytest --cov diofant -m 'not slow' diofant/tests/core

My HTML report looks like this:

We seem to be running different tests.

[skirpichev]

include the steps you used to produce the results you are showing

Sorry, I thought I did it before.

And can you say specifically what about that output is incorrect?

Exactly same as in the issue description, 1st point.

We seem to be running different tests.

I see no difference for L1032 and L1037.

The source of discrepancy seems to be that I did tests with -m 'not slow and not xfail and not regression' instead. That's not an important thing: mentioned before steps should cover exit's from L1032 and L1037. (On the current diofant/diofant#1162 you can run pytest -k test_Pow_is_algebraic --cov diofant diofant/tests/core/test_assumptions.py to trigger these exits.)

[nedbat]

I mistakenly skipped the step of generating the HTML report after running the tests. When I re-do the HTML report, I get this:

[skirpichev]

Unfortunately, I run this step. Are you sure you didn't run tests on CPython 3.9 or more older?

I've tested by pytest -k test_Pow_is_algebraic --cov diofant diofant/tests/core/test_assumptions.py && coverage html on the current diofant/diofant#1162 (merge commit: diofant/diofant@e62cf84). Both for 3.9 and 3.10 CPython (heads/3.10:d5c8ad2471). I still see uncovered exits on L1032 and L1037, as it shown before.

Probably, this is my fault, but I've no glue on what can be wrong on my side.

[skirpichev]

After adding "pass" statements I see this:

for latest CPython 3.10:

$ python
Python 3.10.0rc1+ (heads/3.10:a3185da26f, Aug 10 2021, 06:42:46) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

and

pytest -k test_Pow_is_algebraic --cov diofant diofant/tests/core/test_assumptions.py && coverage html

PS: coveragepy version as before.

[nedbat]

Are you in a virtualenv? The python command may be finding the latest Python 3.10, but the pytest command may be finding a different python. Try using python -m pytest instead.

[skirpichev]

Are you in a virtualenv?

No, I use venv.

Try using python -m pytest instead.

No, both pytest & python -m pytest - report same version.

It seems, git clean -Xdf did the trick. I guess, some old *.pyc files were used.

Thank you very much for help! 1st issue is solved.

What about the second one? I think, the original example may be a non-issue (or a documentation issue), but another one (with a NotImplementedError, which should be ignored) - is.

[nedbat]

@skirpichev I am going to close this issue as solved for #1. Please open a new issue for #2, with specific instructions about the tests you ran to get the output you saw, and specific details about exactly what you think is wrong. I tried running tests to reproduce #2, and got a completely different result than you showed.

Again, I really appreciate your persistence, but we have to get a smoother flow here with reproducible instructions. Assume I know nothing. Give me exact commands to run. Thanks.

[nedbat]

This is now released as part of coverage 6.0.