New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
12061 make twisted.internet.test.test_inlinecb.py compatible with Python 3.13 #12092
base: trunk
Are you sure you want to change the base?
Conversation
please review |
Thansk for the PR. Do you have time to help reviewing this PR first #12059 ? In that PR, python 3.13 automated tests are introduced, so that in future PR we can not only apply changes, but make sure they work on 3.13. Thanks! |
Sure, I can take a look this evening.
…On Tue, Jan 30, 2024, 08:31 Adi Roiban ***@***.***> wrote:
Thansk for the PR.
Do you have time to help reviewing this PR first #12059
<#12059> ?
In that PR, python 3.13 automated tests are introduced, so that in future
PR we can not only apply changes, but make sure they work on 3.13.
Thanks!
—
Reply to this email directly, view it on GitHub
<#12092 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHKYUAMZVOBDYC33KZKILDYRD73HAVCNFSM6AAAAABCJ3A3SGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGAYDCNBVGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Thanks Heather for the PR. I will give it try. As part of this PT, we will also need to update the set of tests we run on Python 3.13 I see there is a FIXME for this Issue to enable more tests It looks like with this change, we should be able to run all the internet tests.
https://github.com/twisted/twisted/blob/trunk/.github/workflows/test.yaml#L146-L150 |
Please call me Weezel. :) |
6cc34a2
to
14df00b
Compare
Thanks Weezel for the update and your help with Python 3.13 support. Which test is failing on your local dev system? Do you use tox or you run For the GitHub Actions job, we use tox, as a way of "standardised" run environment. It's important to have the test pass on local dev environments, but some test are harder to write to account for all the local variations. |
self.assertIn("in calling2", tb) | ||
self.assertIn("in calling3", tb) | ||
self.assertNotIn("throwExceptionIntoGenerator", tb) | ||
self.assertIn("calling2", tb) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't had time to check what is going on here... but at first view, this test change doesn't look ok.
It just removes the check...and is similar to not having a test at all.
If it's easier to implement, we can have 2 separate tests.
One for 3.13 and newer and another for 3.12 and older.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've split these out into Python 3.13 and Python 3.12 and below. Still looking into why this check is saying my tests aren't covered by tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still looking into why this check is saying my tests aren't covered by tests.
Thanks for the tests.
On Python 3.13 we don't have test coverage reporting.
Last time I tried, coverage.py
was not yet supported on Python 3.13 nedbat/coveragepy#1682
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also... slipcover doesn't yet support Python 3.13 ... so for 3.13 we are left uncovered :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Many thanks for the changes.
It looks much better.
Can you please check what happended with the trace for calling3
on Python 3.13.
I don't see any assertion for it.
If this is somehow a "feature", please add a comment in the test to explain why calling3 is not in the stacktrace...also, add an explicit assertion to check that calling3 is missing.... if calling3
is expected to be missing.
Thanks again.
f = self.failureResultOf(d) | ||
tb = f.getTraceback() | ||
self.assertIn("in calling", tb) | ||
self.assertIn("yield calling2", tb) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what happens in Python 3.13, why we don't see the trace for calling3
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is just another change in traceback formatting (like the last 2 python releases), but not indicative of a change in behavior. When running it with a debugger, I see the entire callback chain, but only calling and calling2 (the start of the chain and its callback) included in the traceback text.
Traceback text (apologies for the formatting):
Traceback (most recent call last):
File "/home/eevel/twisted/src/twisted/python/util.py", line 955, in runWithWarningsSuppressed
return f(*args, **kwargs)
File "/home/eevel/twisted/src/twisted/internet/test/test_inlinecb.py", line 265, in test_forwardLotsOfTracebacks_313
d = calling()
File "/home/eevel/twisted/src/twisted/internet/defer.py", line 2260, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/eevel/twisted/src/twisted/internet/defer.py", line 2172, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
File "/home/eevel/twisted/src/twisted/internet/defer.py", line 1999, in _inlineCallbacks
result = context.run(
File "/home/eevel/twisted/src/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
return g.throw(self.value.with_traceback(self.tb))
File "/home/eevel/twisted/src/twisted/internet/test/test_inlinecb.py", line 263, in calling
yield calling2()
builtins.Exception: Error Marker
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added an assertion that calling3 is missing, and updated the docstring to reflect this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have reformated your example.
I am not sure what is going on here.
I see that the erroring
call is also missing.
It looks like Traceback now only shows the top 2 stacks... I don't know.
I don't see anyting about this mentioned in the release notes https://docs.python.org/3.13/whatsnew/3.13.html
Maybe this is a bug in Python 3.13 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I put a traceback at File "/home/eevel/twisted/src/twisted/internet/defer.py", line 1999, in _inlineCallbacks
I can see the full stack... so this looks like a bug in Twisted.
(Pdb) print(result.getBriefTraceback())
Traceback: <class 'Exception'>: Error Marker
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2004:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:257:calling3
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2261:unwindGenerator
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2173:_cancellableInlineCallbacks
--- <exception caught here> ---
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2004:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:253:erroring
(Pdb) c
> /home/adi/chevah/twisted/src/twisted/internet/defer.py(2000)_inlineCallbacks()
-> result = context.run(
(Pdb) print(result.getBriefTraceback())
Traceback: <class 'Exception'>: Error Marker
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2004:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:261:calling2
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2261:unwindGenerator
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2173:_cancellableInlineCallbacks
--- <exception caught here> ---
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2000:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/python/failure.py:519:throwExceptionIntoGenerator
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:257:calling3
(Pdb) c
> /home/adi/chevah/twisted/src/twisted/internet/defer.py(2000)_inlineCallbacks()
-> result = context.run(
(Pdb) print(result.getBriefTraceback())
Traceback: <class 'Exception'>: Error Marker
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2004:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:265:calling1
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2261:unwindGenerator
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2173:_cancellableInlineCallbacks
--- <exception caught here> ---
/home/adi/chevah/twisted/src/twisted/internet/defer.py:2000:_inlineCallbacks
/home/adi/chevah/twisted/src/twisted/python/failure.py:519:throwExceptionIntoGenerator
/home/adi/chevah/twisted/src/twisted/internet/test/test_inlinecb.py:261:calling2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still investigating this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still digging, but I think this might be related to Traceback.exc_type vs Traceback.exc_type_str.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am ruling out TracebackException.exc_type vs exc_type_str as a cause. After digging around in pdb, the codepath appears to be identical, as are the results, until it starts retrieving lines from py3/lib/linecache.getline(). Here are some screenshots of the BdbQuit traceback in 3.12 and 3.13. The 3.13 tb is more sparse overall, and color-highlighted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated the docstring for the 3.13 test to make the reason for this change more apparent, and changed the name of the first callback in chain for the test_forwardLotsOfTracebacks_* tests to "calling1."
needs-review |
raise Exception("Error Marker") | ||
|
||
@inlineCallbacks | ||
def calling(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this needs to be called calling1
or something more specific.
Otherwise self.assertIn("in calling", tb)
will match in callingANYTHIG
and better to use ``self.assertIn("in calling1\n", tb)` to make sure no extra text is added
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the single callback test.
I am not sure what is going on here... Not sure if this is a Twisted bug for 3.13 ...or Python 3.13... or something else. i can see the full stack for non-generators
|
There is also another issue. On 3.12, all tests are skipped
we need something like this ... so that we have the negation of the exact same condition. diff --git a/src/twisted/internet/test/test_inlinecb.py b/src/twisted/internet/test/test_inlinecb.py
index e730e3daf7..36d4b620cf 100644
--- a/src/twisted/internet/test/test_inlinecb.py
+++ b/src/twisted/internet/test/test_inlinecb.py
@@ -127,7 +127,7 @@ class NonLocalExitTests(TestCase):
class ForwardTraceBackTests(SynchronousTestCase):
- @skipIf(sys.version_info > (3, 12), "Needs Python 3.12 or older")
+ @skipIf(not sys.version_info < (3, 13), "Needs Python 3.12 or older")
def test_forwardTracebacks_312(self):
"""
Chained inlineCallbacks are forwarding the traceback information
@@ -177,7 +177,7 @@ class ForwardTraceBackTests(SynchronousTestCase):
self.assertIn("in calling", tb)
self.assertIn("Error Marker", tb)
- @skipIf(sys.version_info > (3, 12), "Needs Python 3.12 or older")
+ @skipIf(not sys.version_info < (3, 13), "Needs Python 3.12 or older")
def test_forwardLotsOfTracebacks_312(self): |
Something even better +
+HAVE_PY3_12_OR_OLDER = sys.version_info < (3, 13)
class ForwardTraceBackTests(SynchronousTestCase):
- @skipIf(sys.version_info > (3, 12), "Needs Python 3.12 or older")
+ @skipIf(not HAVE_PY3_12_OR_OLDER, "Needs Python 3.12 or older")
def test_forwardTracebacks_312(self):
"""
Chained inlineCallbacks are forwarding the traceback information
@@ -152,7 +154,7 @@ class ForwardTraceBackTests(SynchronousTestCase):
self.assertIn("in calling", tb)
self.assertIn("Error Marker", tb)
- @skipIf(sys.version_info < (3, 13), "Needs Python 3.13 or newer")
+ @skipIf(HAVE_PY3_12_OR_OLDER, "Needs Python 3.13 or newer")
def test_forwardTracebacks_313(self):
"""
Chained inlineCallbacks are forwarding the traceback information
@@ -177,7 +179,7 @@ class ForwardTraceBackTests(SynchronousTestCase):
self.assertIn("in calling", tb)
self.assertIn("Error Marker", tb)
- @skipIf(sys.version_info > (3, 12), "Needs Python 3.12 or older")
+ @skipIf(not HAVE_PY3_12_OR_OLDER, "Needs Python 3.12 or older")
def test_forwardLotsOfTracebacks_312(self):
"""
Several Chained inlineCallbacks gives information about all generators.
@@ -225,7 +227,7 @@ class ForwardTraceBackTests(SynchronousTestCase):
self.assertIn("Error Marker", tb)
self.assertIn("in erroring", f.getTraceback())
- @skipIf(sys.version_info < (3, 13), "Needs Python 3.13 or newer")
+ @skipIf(HAVE_PY3_12_OR_OLDER, "Needs Python 3.13 or newer") |
Co-authored-by: Adi Roiban <adiroiban@gmail.com>
Co-authored-by: Adi Roiban <adiroiban@gmail.com>
Co-authored-by: Adi Roiban <adiroiban@gmail.com>
@eevelweezel Thanks for working on this! Is it ready for review again? |
Yes, it is.
…On Mon, Mar 4, 2024, 09:14 Itamar Turner-Trauring ***@***.***> wrote:
@eevelweezel <https://github.com/eevelweezel> Thanks for working on this!
Is it ready for review again?
—
Reply to this email directly, view it on GitHub
<#12092 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHKYUBGSIT7MQHEPZAJEG3YWSFU3AVCNFSM6AAAAABCJ3A3SGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZWHAYTGNBUGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Ugh, blocked by the Cython incompatibility. 3.13alpha5 will be out in a week and a bit, so we should update to that when it's released (it will fix the Cython issue I believe) and then we can see if it's still segfaulting. |
I was looking at #12060 and #12099, and stumbled onto python/cpython#111459, which I think may be preventing failure.Failure._findFailure() from finding the expected information. At least, this seems to be the case for twisted.test.test_failure.ExtendedGeneratorTests.test_findFailureInGenerator, still trying to determine if this is also happening here. My apologies for asking for review prematurely. |
Alright, I have verified that this is where it's breaking, both here and in #12099. On Python3.12, the last instruction executed is YIELD_VALUE:
On Python3.13, the last instruction is RAISE_VARARGS:
I'm still investigating why. |
I've rerun a bunch of the test scenarios locally with both trunk and this branch on 3.13a5. The segfault on twisted.internet is fixed, but now there's a new segfault in twisted.test. However, the last instruction issue is the same for both twisted.internet and twisted.test: |
It does seem like segfaults are entirely CPython's issue; did you report the new one upstream already? |
I haven't yet, but plan to once I have a chance to investigate a bit
further.
…On Thu, Mar 14, 2024 at 12:16 AM Glyph ***@***.***> wrote:
The segfault on twisted.internet is fixed, but now there's a new segfault
in twisted.test
It does seem like segfaults are entirely CPython's issue; did you report
the new one upstream already?
—
Reply to this email directly, view it on GitHub
<#12092 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHKYUA7CE6VFANHVHDMIXTYYEXCFAVCNFSM6AAAAABCJ3A3SGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJWGUZTQMRWGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
______________________________
I should only believe in a God that would know how to dance.
And when I saw my devil, I found him serious, thorough, profound, solemn:
he was the spirit of gravity—through him all things fall. Not by wrath,
but by laughter, do we slay. Come, let us slay the spirit of gravity!
http://4umi.com/nietzsche/zarathustra/7
|
Thanks, really appreciate your work here |
This PR should address the segfault, cython/cython#6084. I'll try again once this merges. This looks relevant as well, python/cpython#108668. |
Retested with 3.13.0a6, and still seeing the same problems. The segfault under twisted.test is still there, but that's to be expected, as the cython PR still hasn't merged. |
Scope and purpose
Fixes #12061 - make twisted.internet.test.test_inlinecb.py compatible with Python 3.13
Fixes #12062
Contributor Checklist:
This process applies to all pull requests - no matter how small.
Have a look at our developer documentation before submitting your Pull Request.
Below is a non-exhaustive list (as a reminder):
please review
.Our bot will trigger the review process, by applying the pending review label
and requesting a review from the Twisted dev team.