Test and support PyPy 3.7

[twisted-trac]

	@altendky reported
Trac ID	trac#10093
Type	enhancement
Created	2021-02-15 17:00:34Z

Let's support the latest PyPy release and Python version. There is one issue described below that needs to be fixed.

The checking of use of t.i.d.returnValue() outside of @inlineCallbacks decorated functions seems to be broken in PyPy 3.7. This was noticed while testing pytest-twisted with pip_pre=true in tox.ini to exercise it against the Twisted 21.2.0rc1 pre-release.

pytest-twisted CI

https://github.com/pytest-dev/pytest-twisted/pull/134/checks?check_run_id=1904583062

E                   DeprecationWarning: returnValue() in '_async_pytest_pyfunc_call' causing 'run' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks

But...

https://github.com/pytest-dev/pytest-twisted/blob/510c02cb9cb67df4066b837ce917c2e8b1196989/pytest_twisted.py#L361-L362

@defer.inlineCallbacks
def _async_pytest_pyfunc_call(pyfuncitem):

Twisted CI with PyPy 3.7

https://github.com/twisted/twisted/pull/1515/checks?check_run_id=1904809978

  NonLocalExitTests
    test_returnValueNonLocalDeferred ...                                 [FAIL]
    test_returnValueNonLocalWarning ...                                  [FAIL]

2021-02-15T16:42:42.5345068Z twisted.trial.test.test_assertions.ResultOfCoroutineAssertionsTests.test_successResultOfWithFailureHasTraceback
2021-02-15T16:42:42.5346930Z ===============================================================================
2021-02-15T16:42:42.5347243Z [FAIL]
2021-02-15T16:42:42.5347940Z Traceback (most recent call last):
2021-02-15T16:42:42.5349674Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/internet/test/test_inlinecb.py", line 125, in test_returnValueNonLocalDeferred
2021-02-15T16:42:42.5351341Z     self.assertMistakenMethodWarning(results)
2021-02-15T16:42:42.5353107Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/internet/test/test_inlinecb.py", line 84, in assertMistakenMethodWarning
2021-02-15T16:42:42.5354741Z     "returnValue() in 'mistakenMethod' causing 'inline' to exit: "
2021-02-15T16:42:42.5356396Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/trial/_synctest.py", line 424, in assertEqual
2021-02-15T16:42:42.5357436Z     super().assertEqual(first, second, msg)
2021-02-15T16:42:42.5358585Z   File "/opt/hostedtoolcache/PyPy/3.7.9/x64/lib-python/3/unittest/case.py", line 852, in assertEqual
2021-02-15T16:42:42.5359283Z     assertion_func(first, second, msg=msg)
2021-02-15T16:42:42.5362571Z   File "/opt/hostedtoolcache/PyPy/3.7.9/x64/lib-python/3/unittest/case.py", line 1233, in assertMultiLineEqual
2021-02-15T16:42:42.5363434Z     self.fail(self._formatMessage(msg, standardMsg))
2021-02-15T16:42:42.5366382Z twisted.trial.unittest.FailTest: "retu[34 chars]ing 'run' to exit: returnValue should only be [47 chars]acks" != "retu[34 chars]ing 'inline' to exit: returnValue should only [50 chars]acks"
2021-02-15T16:42:42.5368958Z - returnValue() in 'mistakenMethod' causing 'run' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks
2021-02-15T16:42:42.5370020Z ?                                            ^^
2021-02-15T16:42:42.5371928Z + returnValue() in 'mistakenMethod' causing 'inline' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks
2021-02-15T16:42:42.5372659Z ?                                            ^ ++++
2021-02-15T16:42:42.5372867Z 
2021-02-15T16:42:42.5373039Z 
2021-02-15T16:42:42.5373872Z twisted.internet.test.test_inlinecb.NonLocalExitTests.test_returnValueNonLocalDeferred
2021-02-15T16:42:42.5374764Z ===============================================================================
2021-02-15T16:42:42.5375419Z [FAIL]
2021-02-15T16:42:42.5376162Z Traceback (most recent call last):
2021-02-15T16:42:42.5378682Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/internet/test/test_inlinecb.py", line 104, in test_returnValueNonLocalWarning
2021-02-15T16:42:42.5380138Z     self.assertMistakenMethodWarning(results)
2021-02-15T16:42:42.5382137Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/internet/test/test_inlinecb.py", line 84, in assertMistakenMethodWarning
2021-02-15T16:42:42.5384773Z     "returnValue() in 'mistakenMethod' causing 'inline' to exit: "
2021-02-15T16:42:42.5386476Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/trial/_synctest.py", line 424, in assertEqual
2021-02-15T16:42:42.5387924Z     super().assertEqual(first, second, msg)
2021-02-15T16:42:42.5389059Z   File "/opt/hostedtoolcache/PyPy/3.7.9/x64/lib-python/3/unittest/case.py", line 852, in assertEqual
2021-02-15T16:42:42.5389972Z     assertion_func(first, second, msg=msg)
2021-02-15T16:42:42.5392302Z   File "/opt/hostedtoolcache/PyPy/3.7.9/x64/lib-python/3/unittest/case.py", line 1233, in assertMultiLineEqual
2021-02-15T16:42:42.5393813Z     self.fail(self._formatMessage(msg, standardMsg))
2021-02-15T16:42:42.5395523Z twisted.trial.unittest.FailTest: "retu[34 chars]ing 'run' to exit: returnValue should only be [47 chars]acks" != "retu[34 chars]ing 'inline' to exit: returnValue should only [50 chars]acks"
2021-02-15T16:42:42.5397835Z - returnValue() in 'mistakenMethod' causing 'run' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks
2021-02-15T16:42:42.5398739Z ?                                            ^^
2021-02-15T16:42:42.5399681Z + returnValue() in 'mistakenMethod' causing 'inline' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks
2021-02-15T16:42:42.5400617Z ? 

Searchable metadata

trac-id__10093 10093
type__enhancement enhancement
reporter__altendky altendky
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__closed closed
resolution__fixed fixed
component__core core
keywords__review review
time__1613408434347621 1613408434347621
changetime__1613437778775512 1613437778775512
version__None None
owner__Kyle_Altendorf__sda_____ Kyle Altendorf <sda@...>

[twisted-trac]

	@altendky set owner to @altendky

[twisted-trac]

	@altendky commented

PR: #1516

[twisted-trac]

	@rodrigc commented

I was able to repro the problem by doing the following.

1. I set up a Fedora 33 VM
2. git clone https://github.com/pytest-dev/pytest-twisted/
3. git clone https://github.com/twisted/twisted
4. Download build from https://buildbot.pypy.org/nightly/py3.8/
5. Apply this patch to pypy:

--- pypy-c-jit-101599-50248e277620-linux64/lib-python/3/venv/__init__.py.orig   2021-02-15 22:26:53.247408936 +0000
+++ pypy-c-jit-101599-50248e277620-linux64/lib-python/3/venv/__init__.py        2021-02-15 22:25:12.942950266 +0000
@@ -105,7 +105,7 @@
         prompt = self.prompt if self.prompt is not None else context.env_name
         context.prompt = '(%s) ' % prompt
         create_if_needed(env_dir)
-        executable = sys._base_executable
+        executable = sys.executable
         dirname, exename = os.path.split(os.path.abspath(executable))
         context.executable = executable
         context.python_dir = dirname

6. pypy3 -m ensurepip
7. pypy3 -m venv venv-twisted
8. source venv-twsited/bin/activate
9. cd $HOME/twisted
10. python -m pip install incremental
11. python setup.py develop
12. cd $HOME/pytest-twisted/
13. python setup.py develop
14. pytest -v -v --reactor=asyncio testing/test_basic.py::test_ensuredeferred_method_with_fixture_gets_self

I got:

-------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------
running: /home/vagrant/venv-twisted-pypy3/bin/python -m pytest -v --reactor=asyncio
     in: /tmp/pytest-of-vagrant/pytest-19/test_ensuredeferred_method_with_fixture_gets_self0
============================= test session starts ==============================
platform linux -- Python 3.8.7[pypy-7.3.4-alpha], pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/vagrant/venv-twisted-pypy3/bin/python
cachedir: .pytest_cache
rootdir: /tmp/pytest-of-vagrant/pytest-19/test_ensuredeferred_method_with_fixture_gets_self0
plugins: twisted-1.13.2
collecting ... collected 1 item

test_ensuredeferred_method_with_fixture_gets_self.py::TestClass::test_self_isinstance PASSED [100%]

=============================== warnings summary ===============================
test_ensuredeferred_method_with_fixture_gets_self.py::TestClass::test_self_isinstance
  /home/vagrant/pytest-twisted/pytest_twisted.py:379: DeprecationWarning: returnValue() in '_async_pytest_pyfunc_call' causing 'run' to exit: returnValue should only be invoked by functions decorated with inlineCallbacks
    defer.returnValue(result)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================= 1 passed, 1 warning in 0.04s =========================
============================================================================== short test summary info ===============================================================================
FAILED testing/test_basic.py::test_ensuredeferred_method_with_fixture_gets_self - AssertionError:

[twisted-trac]

	@rodrigc commented

I don't know if this is the right way to go, but by applying this:

diff --git a/src/twisted/internet/defer.py b/src/twisted/internet/defer.py
index f9c6dd335..bb1643d17 100644
--- a/src/twisted/internet/defer.py
+++ b/src/twisted/internet/defer.py
@@ -31,7 +31,7 @@ from incremental import Version
 # Twisted imports
 from twisted.logger import Logger
 from twisted.python import failure, lockfile
-from twisted.python.compat import cmp, comparable
+from twisted.python.compat import cmp, comparable, _PYPY
 from twisted.python.deprecate import deprecated, warnAboutFunction

 try:
@@ -1473,7 +1473,7 @@ def _inlineCallbacks(result, g, status):
             # directly.  returnValue itself consumes a stack frame, so the
             # application code will have a tb_next, but it will *not* have a
             # second tb_next.
-            if appCodeTrace.tb_next.tb_next:
+            if appCodeTrace.tb_next.tb_next and not _PYPY:
                 # If returnValue was invoked non-local to the frame which it is
                 # exiting, identify the frame that ultimately invoked
                 # returnValue so that we can warn the user, as this behavior is

Running this:
pytest -v -v --reactor=asyncio testing/test_basic.py::test_ensuredeferred_method_with_fixture_gets_self

would pass without any warnings

[twisted-trac]

	@altendky removed owner

[twisted-trac]

	@altendky set owner to @altendky @altendky set status to closed

In changeset a72db25

#!CommitTicketReference repository="" revision="a72db25e474db26765d7eb1607b654cb14934ae5"
Merge pull request #1516 from altendky/altendky-10093-returnValue_DeprecationWarning_pypy37

Merge pull request #1516 from altendky/altendky-10093-returnValue_DeprecationWarning_pypy37

Author: altendky
Reviewer: rodrigc
Fixes: ticket:10093

This adds PyPy 3.7 to the test matrix and addresses a false positive returnValue() DeprecationWarning when running in PyPy 3.7. This showed up originally in pytest-twisted tests.