Fix Python 3.10 test issues #8555

The-Compiler · 2021-04-14T09:40:47Z

Potential alternative to #8553. Fixes the Python 3.10 tests for me, without #8540. See #8546, though I wouldn't close it yet, as we might want to change the enum reprs instead?

The-Compiler · 2021-04-14T10:02:25Z

Huh. ubuntu-py310 works, but on windows-py310, all tests using @pytest.mark.filterwarnings("default") (I think?) still fail. I'm stumped.

testing/python/metafunc.py

testing/test_pytester.py

The-Compiler · 2021-04-15T18:15:10Z

Fair point @nicoddemus - but do you have any idea what's up with the failures on Windows?

nicoddemus · 2021-04-15T18:53:04Z

Fair point @nicoddemus - but do you have any idea what's up with the failures on Windows?

Ahh sorry, missed those comments. Not at a quick glance, will try to test this locally when I get the chance. 👍

bluetech · 2021-04-20T13:29:45Z

testing/test_pytester.py

-        repr(r) == "<RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=3"
-        " len(stderr.lines)=4 duration=0.50s>"
-    )
+    expected = [


I suggest we just make it consistent here -- make it use one or the other (the new one looks better to me)

I suppose we definitely will search for "3.10" when dropping that version, so in the end doesn't really matter I guess.

bluetech · 2021-04-20T13:31:02Z

testing/python/metafunc.py

@@ -448,7 +448,10 @@ def test_idmaker_enum(self) -> None:
        enum = pytest.importorskip("enum")
        e = enum.Enum("Foo", "one, two")
        result = idmaker(("a", "b"), [pytest.param(e.one, e.two)])
-        assert result == ["Foo.one-Foo.two"]
+        assert result in [


I described my preferred solution here, though shouldn't hold up this PR.

nicoddemus · 2021-04-24T13:48:44Z

OK did some debugging (sorry for the delay) and here is what I found.

First I could not reproduce the tests running them in isolation, I needed to do a binary search and to find the minimal two tests that when executed in order, would fail the second:

import pytest

pytest.main([
	"testing/test_nose.py::test_raises",
	"testing/acceptance_test.py::test_warn_on_async_function", 
	"-x", 
	"-v"],
)

Running this script causes test_warn_on_async_function to fail, which it doesn't when run in isolation.

Looking at test_nose.py::test_raises, we don't see anything special in there, so I suspected that just by importing nose we were triggering the problem. I changed the test to:

def test_raises(pytester: Pytester) -> None:
    pytester.makepyfile("import nose")

And indeed the problem happens, so importing nose is triggering the extra warning in test_warn_on_async_function.

Why it fails? Well it changes the warning filtering to default:

@pytest.mark.filterwarnings("default")
def test_warn_on_async_function(pytester: Pytester) -> None:

And it checks the number of warnings exactly:

	result.stdout.fnmatch_lines(
        [
            "test_async.py::test_1",
            "test_async.py::test_2",
            "test_async.py::test_3",
            "*async def functions are not natively supported*",
            "*3 skipped, 3 warnings in*",
        ]
    )

So when nose is imported, an extra warning is triggered, which causes that check to fail (expected 3 warnings, got 4).

Why this is happening only on Windows? I suspect there's some Windows-specific code which causes that warning to be triggered, but I didn't research it further.

Why is the warning leaking from one test to the other? Not sure too.

What to do?

I think the test is a little brittle, it expects only 3 warnings even though it changed the warnings filter to default, so any extra warnings raised in the future would cause it to fail. I think we don't need to check the exact number of warnings here, just checking the messages is enough, so perhaps this:

	result.stdout.fnmatch_lines(
        [
            "test_async.py::test_1",
            "test_async.py::test_2",
            "test_async.py::test_3",
            "*async def functions are not natively supported*",
            "*3 skipped, * warnings in*",
        ]
    )

?

Didn't check the other tests specifically, but I suspect it is the same problem.

bluetech · 2021-04-25T07:35:06Z

What to do?

I agree with your suggestion.

The-Compiler · 2021-04-28T06:09:50Z

Sounds good @nicoddemus. Still busy with my pytest training, but I plan to pick this up tomorrow or on Friday. I think it's really important to have a passing CI again before everyone starts ignoring (or being confused by) the commit status.

nicoddemus · 2021-04-28T10:53:20Z

Definitely, thanks for tackling this @The-Compiler!

The-Compiler · 2021-04-29T16:05:44Z

Looks like there are way more tests failing though, sometimes we're e.g. expecting a certain order in the warning output, but the new warnings mess everything up. Still no idea why that's only the case on Windows, though.

Will need to dig into it some more I guess, but not today.

benjaminp/six#341 benjaminp/six#352 pypa/setuptools#2517

Potential fix for pytest-dev#8546

The-Compiler · 2021-05-04T15:33:45Z

This should be ready for another round of review now - and hopefully finally result in a green CI everywhere, fingers crossed! 🤞

A couple of comments:

Agreed with making the version checks more explicit, based on @nicoddemus' suggestions.
I disagree with @nicoddemus' proposal above on how to handle warnings in tests. I don't think we should write our integration tests in a way that they are less strict and pass even if other warnings occur - in fact, quite the opposite! If unexpected warnings occur, the tests should fail. So instead of changing the matching, I decided to adjust the filterwarnings marks so they don't unconditionally set default (or always) for all warnings (which would ignore our existing filters in pyproject.toml, and also ignore the default error filter, i.e. potentially hide real issues!). Instead, they now only affect the warning class the test in question actually uses.
test_collect_symlink_dir failed for me on Windows (which needs special privileges to create symlinks). Not sure how it ever could've run in earlier versions, as it didn't use symlink_or_skip like all the others. Maybe it just runs on CI because the CI has those privileges or something. I didn't investigate further, as this PR already took a lot more work than expected 😅
The beta 1 brought a couple of new issues, which now should all be fixed as well. One of them is Update threading.Event.isSet() calls execnet#127, the other is changed output for SyntaxErrors in skip/xfail expressions:

_______________________ ERROR at setup of test_nameerror _______________________
name 'asd' is not defined

During handling of the above exception, another exception occurred:
Error evaluating 'skipif' condition
    asd
NameError: name 'asd' is not defined
________________________ ERROR at setup of test_syntax _________________________
invalid syntax. Perhaps you forgot a comma? (<xfail condition>, line 1)

During handling of the above exception, another exception occurred:
Error evaluating 'xfail' condition
    syntax error
     ^
SyntaxError: invalid syntax

(the "During handling of the above exception, another exception occurred:" and everything above it is new).

The new output maybe isn't optimal from an user perspective, but if we want to change that, I suggest to open a separate issue.

I also rebased and cleaned up the commits - I'd suggest not squashing this as there are different kinds of changes.

nicoddemus

Awesome, thanks!

The-Compiler · 2021-05-05T12:21:32Z

Taking the freedom to merge this, since it's approved and I'd really like to see passing PR status checks again. @bluetech if you feel strongly about changing pytest to always output consistent enum repr's, I suggest you open a new issue. I can see where you're coming from, but personally I think it's too minor to worry about it, as those repr's aren't typically exposed to users.

Fix Python 3.10 test issues (cherry picked from commit adc1974)

hroncok · 2021-07-11T23:54:05Z

Adjust enum reprs for Python 3.10

Note that the Enum reprs change was reverted in upstream.

https://mail.python.org/archives/list/python-dev@python.org/message/LSTMFAPSPD3BGZ4D6HQFODXZVB3PLYKF/

The Python version with the reverted thing is 3.10.0b4. We can adapt the if to say 3.11, or we can revert it for now, or we can adapt the if to be more complex (and check for the actual pre-release).

nicoddemus · 2021-07-12T13:15:51Z

We can adapt the if to say 3.11, or we can revert it for now, or we can adapt the if to be more complex (and check for the actual pre-release).

I vote to adapt the if to use 3.11.

hroncok · 2021-07-12T13:49:10Z

On it.

hroncok · 2021-07-12T13:52:32Z

#8895

The-Compiler requested a review from hroncok April 14, 2021 09:40

nicoddemus reviewed Apr 15, 2021

View reviewed changes

testing/python/metafunc.py Outdated Show resolved Hide resolved

nicoddemus reviewed Apr 15, 2021

View reviewed changes

testing/test_pytester.py Outdated Show resolved Hide resolved

bluetech reviewed Apr 20, 2021

View reviewed changes

The-Compiler added 4 commits May 4, 2021 17:24

Ignore various warnings from Python 3.10

f1570dc

benjaminp/six#341 benjaminp/six#352 pypa/setuptools#2517

Adjust enum reprs for Python 3.10

7104464

Potential fix for pytest-dev#8546

Fix warning filters used in tests

e354c5c

Fix test_collect_symlink_dir on Windows

61be48b

The-Compiler force-pushed the py310-fix branch from 248da78 to 529fa29 Compare May 4, 2021 15:25

Fix test_errors_in_xfail_skip_expressions on Python 3.10

3825992

The-Compiler force-pushed the py310-fix branch from 6e30e91 to 3825992 Compare May 4, 2021 15:27

The-Compiler requested review from nicoddemus and bluetech May 4, 2021 15:35

nicoddemus approved these changes May 4, 2021

View reviewed changes

The-Compiler merged commit adc1974 into pytest-dev:main May 5, 2021

mgorny pushed a commit to mgorny/pytest that referenced this pull request May 5, 2021

Merge pull request pytest-dev#8555 from The-Compiler/py310-fix

1e1955d

Fix Python 3.10 test issues (cherry picked from commit adc1974)

nicoddemus mentioned this pull request Jul 12, 2021

Closes #8796 - DoctTest line numbers not found due to method wrapping #8861

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Python 3.10 test issues #8555

Fix Python 3.10 test issues #8555

The-Compiler commented Apr 14, 2021

The-Compiler commented Apr 14, 2021

The-Compiler commented Apr 15, 2021

nicoddemus commented Apr 15, 2021

bluetech Apr 20, 2021

nicoddemus Apr 24, 2021

bluetech Apr 20, 2021

nicoddemus commented Apr 24, 2021 •

edited

bluetech commented Apr 25, 2021

The-Compiler commented Apr 28, 2021

nicoddemus commented Apr 28, 2021

The-Compiler commented Apr 29, 2021

The-Compiler commented May 4, 2021 •

edited

nicoddemus left a comment

The-Compiler commented May 5, 2021

hroncok commented Jul 11, 2021 •

edited

nicoddemus commented Jul 12, 2021

hroncok commented Jul 12, 2021

hroncok commented Jul 12, 2021

Fix Python 3.10 test issues #8555

Fix Python 3.10 test issues #8555

Conversation

The-Compiler commented Apr 14, 2021

The-Compiler commented Apr 14, 2021

The-Compiler commented Apr 15, 2021

nicoddemus commented Apr 15, 2021

bluetech Apr 20, 2021

Choose a reason for hiding this comment

nicoddemus Apr 24, 2021

Choose a reason for hiding this comment

bluetech Apr 20, 2021

Choose a reason for hiding this comment

nicoddemus commented Apr 24, 2021 • edited

bluetech commented Apr 25, 2021

The-Compiler commented Apr 28, 2021

nicoddemus commented Apr 28, 2021

The-Compiler commented Apr 29, 2021

The-Compiler commented May 4, 2021 • edited

nicoddemus left a comment

Choose a reason for hiding this comment

The-Compiler commented May 5, 2021

hroncok commented Jul 11, 2021 • edited

Adjust enum reprs for Python 3.10

nicoddemus commented Jul 12, 2021

hroncok commented Jul 12, 2021

hroncok commented Jul 12, 2021

nicoddemus commented Apr 24, 2021 •

edited

The-Compiler commented May 4, 2021 •

edited

hroncok commented Jul 11, 2021 •

edited