[1.1.2] Demonstrate that C++ exception thrown in one greenlet, can propagate to another greenlet and corrupt memory #285
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…opagate to another greenlet and corrupt memory Hello up there. While working on Pygolang I've faced the following problem: a C++ exception thrown in one greenlet, without any try/catch block on that greenlet, might become propagated and "handled" by try/catch block established at another greenlet that happened to switch to the greenlet in question. "Handled" comes in quotes because the program usually segfaults after that. I've also observed segfaults before exception reaches user-level try/catch block - internally in __cxa_throw. Both this issues are likely of the same origin - due to the fact that C-level stack of a greenlet does not start from scratch and starts from C-level stack of the program state when the greenlet is switched to the first time. We already have one test for C++ exception handling in test_cpp that @snaury initially added in d9cb12a. However in that test all greenlets that throw C++ exceptions also put try/catch at the top of their C-stack. In the problem, that I describe, and that added test reproduces, there is no intended top-level try/catch block in C-stack of the greenlet in question that throws. As the test shows the exception becomes propagated to switcher's greenlet context and the program then dies with SIGSEGV: test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests) ... terminate called after throwing an instance of 'exception_t' C++ exception unexpectedly caught in g1 <-- NOTE FAIL ====================================================================== FAIL: test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/kirr/src/tools/py/gevent/greenlet/src/greenlet/tests/test_cpp.py", line 62, in test_exception_switch_and_throw (ret, sig, " (core dumped)" if core else "")) AssertionError: failed with ret=0 sig=11 (core dumped) The C-level backtrace from the dumped core is attached in Appendix I. There the program dies somehow after running the code from _test_extension_cpp.cpp module. However with the following dirty-patch the situation becomes more clear: --- a/src/greenlet/tests/_test_extension_cpp.cpp +++ b/src/greenlet/tests/_test_extension_cpp.cpp @@ -70,6 +70,7 @@ test_exception_switch(PyObject* self, PyObject* args) static PyObject* py_test_exception_throw(PyObject* self, PyObject* args) { + abort(); if (!PyArg_ParseTuple(args, "")) return NULL; p_test_exception_throw(0); The C-level backtrace of this abort when, run from under g2, shows that g2 C-stack starts from what was there before for first greenlet with try/catch block: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 python-greenlet#1 0x00007f8be31e9537 in __GI_abort () at abort.c:79 python-greenlet#2 0x00007f8be3529423 in py_test_exception_throw (self=0x0, args=0x7f8be317f050) at src/greenlet/tests/_test_extension_cpp.cpp:73 python-greenlet#3 0x00005584d0389903 in PyObject_Call (func=0x7f8be2cdd690, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#4 0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b1c88) at src/greenlet/greenlet.c:931 python-greenlet#5 0x00007f8be3530290 in g_switch (target=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692 python-greenlet#6 0x00007f8be35329e2 in PyGreenlet_Switch (g=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1806 <- first switch to G2 python-greenlet#7 0x00007f8be35294da in test_exception_switch_and_do_in_g2 (self=0x0, args=0x7f8be2ce33d0) <- this code runs in G1 at src/greenlet/tests/_test_extension_cpp.cpp:105 python-greenlet#8 0x00005584d039de7a in call_function (oparg=<optimized out>, pp_stack=0x7ffe6d1b1e28) at ../Python/ceval.c:4376 python-greenlet#9 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#10 0x00005584d039c3cc in PyEval_EvalCodeEx (co=0x7f8be2cd9cb0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#11 0x00005584d03b6b9b in function_call (func=func@entry=0x7f8be2cde650, arg=0x7f8be317f050, kw=0x0) at ../Objects/funcobject.c:523 python-greenlet#12 0x00005584d0389903 in PyObject_Call (func=0x7f8be2cde650, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#13 0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b20b8) at src/greenlet/greenlet.c:931 python-greenlet#14 0x00007f8be3530290 in g_switch (target=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692 python-greenlet#15 0x00007f8be3531c4c in green_switch (self=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1321 <- switch to G1 ... The problem might be worked-around with putting try/catch on C-stack for every greenlet, but a more proper fix would be to unlink return address and C-stack-frame of created greenlet from other greenlets completely (i.e. rewrite first frame of C-stack for created greenlet to return to NULL). Thanks beforehand, Kirill P.S. The problem described at python-greenlet#197 (comment) and python-greenlet#205 might be related to hereby issue (/cc @ThePrez, @kadler, @jamadden). P.P.S. The test does not fail on master. I see there @jamadden switched the codebase to C++ almost completely and there is some explicit attempts to save/restore exception state added in python-greenlet@87edf955. However that exception save/restore is specific to Win32 and the test succeeds for me on Linux. I still see the following comment in UserGreenlet::inner_bootstrap // C++ exceptions cannot propagate to the parent greenlet from // here. (TODO: Do we need a catch(...) clause, perhaps on the // function itself? ALl we could do is terminate the program.) ( https://github.com/python-greenlet/greenlet/blob/3e534d6b/src/greenlet/greenlet.cpp#L1085-L1104 ) probably this indeed works only by luck, or due to G_NOEXCEPT modifier of UserGreenlet::inner_bootstrap upon seeing which __cxa_throw stops unwinding C-stack and just calls std::terminate. -------- Appendix I. C-level backtrace after the test died with SIGSEGV #0 0x0000556bdc30d394 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3351 python-greenlet#1 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03acb0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#2 0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a03f650, arg=0x7f0b8a4e0050, kw=0x0) at ../Objects/funcobject.c:523 python-greenlet#3 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03f650, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#4 0x00007f0b8a891d63 in g_initialstub (mark=0x7ffc3ef97988) at src/greenlet/greenlet.c:931 python-greenlet#5 0x00007f0b8a891290 in g_switch (target=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:692 python-greenlet#6 0x00007f0b8a892c4c in green_switch (self=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:1321 python-greenlet#7 0x0000556bdc30ce7a in call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ac8) at ../Python/ceval.c:4376 python-greenlet#8 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#9 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ad30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#10 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97ca8, func=0x7f0b8a03aed0) at ../Python/ceval.c:4471 python-greenlet#11 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ca8) at ../Python/ceval.c:4396 python-greenlet#12 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#13 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ac30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#14 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97e88, func=0x7f0b8a03af50) at ../Python/ceval.c:4471 python-greenlet#15 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97e88) at ../Python/ceval.c:4396 python-greenlet#16 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#17 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03adb0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#18 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef98068, func=0x7f0b8a03f250) at ../Python/ceval.c:4471 python-greenlet#19 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98068) at ../Python/ceval.c:4396 python-greenlet#20 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#21 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bd30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a0752a8, defcount=1, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#22 0x0000556bdc325cc2 in function_call (func=0x7f0b8a0771d0, arg=0x7f0b8a3bd5a0, kw=0x7f0b8a042dd0) at ../Objects/funcobject.c:523 python-greenlet#23 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042dd0, arg=0x7f0b8a3bd5a0, func=0x7f0b8a0771d0) at ../Objects/abstract.c:2544 python-greenlet#24 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef982a0, func=0x7f0b8a0771d0) at ../Python/ceval.c:4690 python-greenlet#25 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052 python-greenlet#26 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bf30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#27 0x0000556bdc325b9b in function_call (func=0x7f0b8a0772d0, arg=arg@entry=0x7f0b8a3bd550, kw=kw@entry=0x0) at ../Objects/funcobject.c:523 python-greenlet#28 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd550, func=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#29 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6a00, arg=0x7f0b8a3bd550, kw=0x0) at ../Objects/classobject.c:2600 python-greenlet#30 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6a00, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#31 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cfd0, args=0x7f0b8a044210, kwds=0x0) at ../Objects/typeobject.c:5609 python-greenlet#32 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cfd0, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#33 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98798, func=0x7f0b8a03cfd0) at ../Python/ceval.c:4593 python-greenlet#34 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98798) at ../Python/ceval.c:4398 python-greenlet#35 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#36 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#37 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd780, kw=0x7f0b8a042cb0) at ../Objects/funcobject.c:523 python-greenlet#38 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042cb0, arg=0x7f0b8a3bd780, func=0x7f0b8a07f950) at ../Objects/abstract.c:2544 python-greenlet#39 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef989d0, func=0x7f0b8a07f950) at ../Python/ceval.c:4690 python-greenlet#40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052 python-greenlet#41 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#42 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd730, kw=kw@entry=0x0) at ../Objects/funcobject.c:523 python-greenlet#43 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd730, func=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#44 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427190, arg=0x7f0b8a3bd730, kw=0x0) at ../Objects/classobject.c:2600 python-greenlet#45 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a427190, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#46 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cf90, args=0x7f0b8a044150, kwds=0x0) at ../Objects/typeobject.c:5609 python-greenlet#47 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cf90, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#48 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98ec8, func=0x7f0b8a03cf90) at ../Python/ceval.c:4593 python-greenlet#49 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98ec8) at ../Python/ceval.c:4398 python-greenlet#50 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#51 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#52 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bdaf0, kw=0x7f0b8a042b90) at ../Objects/funcobject.c:523 python-greenlet#53 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042b90, arg=0x7f0b8a3bdaf0, func=0x7f0b8a07f950) at ../Objects/abstract.c:2544 python-greenlet#54 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99100, func=0x7f0b8a07f950) at ../Python/ceval.c:4690 python-greenlet#55 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052 python-greenlet#56 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#57 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bdb40, kw=kw@entry=0x0) at ../Objects/funcobject.c:523 python-greenlet#58 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bdb40, func=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#59 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6820, arg=0x7f0b8a3bdb40, kw=0x0) at ../Objects/classobject.c:2600 python-greenlet#60 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6820, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#61 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c50, args=0x7f0b8a03c0d0, kwds=0x0) at ../Objects/typeobject.c:5609 python-greenlet#62 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c50, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#63 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef995f8, func=0x7f0b8a033c50) at ../Python/ceval.c:4593 python-greenlet#64 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef995f8) at ../Python/ceval.c:4398 python-greenlet#65 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#66 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#67 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd500, kw=0x7f0b8a042a70) at ../Objects/funcobject.c:523 python-greenlet#68 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042a70, arg=0x7f0b8a3bd500, func=0x7f0b8a07f950) at ../Objects/abstract.c:2544 python-greenlet#69 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99830, func=0x7f0b8a07f950) at ../Python/ceval.c:4690 python-greenlet#70 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052 python-greenlet#71 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#72 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd9b0, kw=kw@entry=0x0) at ../Objects/funcobject.c:523 python-greenlet#73 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd9b0, func=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#74 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a4310a0, arg=0x7f0b8a3bd9b0, kw=0x0) at ../Objects/classobject.c:2600 python-greenlet#75 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a4310a0, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#76 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c10, args=0x7f0b8a033fd0, kwds=0x0) at ../Objects/typeobject.c:5609 python-greenlet#77 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c10, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#78 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef99d28, func=0x7f0b8a033c10) at ../Python/ceval.c:4593 python-greenlet#79 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99d28) at ../Python/ceval.c:4398 python-greenlet#80 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#81 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99e38, func=0x7f0b8a01ecd0) at ../Python/ceval.c:4461 python-greenlet#82 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99e38) at ../Python/ceval.c:4396 python-greenlet#83 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#84 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99f48, func=0x7f0b8a01edd0) at ../Python/ceval.c:4461 python-greenlet#85 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99f48) at ../Python/ceval.c:4396 python-greenlet#86 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#87 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a081d30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=1, defs=0x7f0b8a008ba8, defcount=10, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#88 0x0000556bdc325cc2 in function_call (func=0x7f0b8a00ddd0, arg=arg@entry=0x7f0b8a033610, kw=kw@entry=0x7f0b8a014a70) at ../Objects/funcobject.c:523 python-greenlet#89 0x0000556bdc33f553 in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a033610, func=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#90 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427140, arg=0x7f0b8a033610, arg@entry=0x7f0b8a4e0050, kw=kw@entry=0x7f0b8a014a70) at ../Objects/classobject.c:2600 python-greenlet#91 0x0000556bdc33f10f in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a4e0050, func=0x7f0b8a427140) at ../Objects/abstract.c:2544 python-greenlet#92 slot_tp_init (self=self@entry=0x7f0b8a3ebb10, args=args@entry=0x7f0b8a4e0050, kwds=kwds@entry=0x7f0b8a014a70) at ../Objects/typeobject.c:5869 python-greenlet#93 0x0000556bdc2feb87 in type_call (type=<optimized out>, type@entry=0x556bdd2a85c0, args=0x7f0b8a4e0050, kwds=0x7f0b8a014a70) at ../Objects/typeobject.c:765 python-greenlet#94 0x0000556bdc2f8903 in PyObject_Call (func=0x556bdd2a85c0, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#95 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef9a458, func=0x556bdd2a85c0) at ../Python/ceval.c:4593 python-greenlet#96 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a458) at ../Python/ceval.c:4398 python-greenlet#97 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#98 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d830, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#99 0x0000556bdc311886 in PyEval_EvalCode (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, co=<optimized out>) at ../Python/ceval.c:669 python-greenlet#100 exec_statement (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, prog=<optimized out>, f=<optimized out>) at ../Python/ceval.c:5093 python-greenlet#101 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:2122 python-greenlet#102 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a3ea4b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a381068, defcount=5, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#103 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef9a818, func=0x7f0b8a37d8d0) at ../Python/ceval.c:4471 python-greenlet#104 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a818) at ../Python/ceval.c:4396 python-greenlet#105 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013 python-greenlet#106 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d4b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a379ce8, defcount=1, closure=0x0) at ../Python/ceval.c:3608 python-greenlet#107 0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a37de50, arg=0x7f0b8a3b5be0, kw=0x0) at ../Objects/funcobject.c:523 python-greenlet#108 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a37de50, arg=<optimized out>, kw=<optimized out>) at ../Objects/abstract.c:2544 python-greenlet#109 0x0000556bdc3b87e1 in RunModule (module=<optimized out>, set_argv0=1) at ../Modules/main.c:197 python-greenlet#110 0x0000556bdc3a68ed in Py_Main (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:592#111 0x00007f0b8a54bd0a in __libc_start_main (main=0x556bdc3a6530 <main>, argc=5, argv=0x7ffc3ef9ac08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc3ef9abf8) at ../csu/libc-start.c:308 python-greenlet#112 0x0000556bdc3a646a in _start ()
|
Thanks for the test! Would you be interested in updating it to work with the main branch? It's a valuable addition to the test suite. (I don't expect to make any exception-related changes to older code. One of the main goals for 2.0 is to improve the situation, and I believe that requires a more modern compiler than was required for 1.x.) |
Sync to 2.0 development where the test does not fail, at least on Linux. * master: (219 commits) Another TODO comment [skip ci] Back to development: 2.0.0a2 Preparing release 2.0.0a1 Update MANIFEST.in to exclude results from benchmarks Update CHANGES for python-greenlet#280. Bump GHA Python 3.11 from a2 to a4 Refactor for NULL datastack support Restore compatibility with Python 3.11 Add back missing members Fix compatibility macro Revert unrelated change Fix exc_state usage for Python 3.11 Add change note for python-greenlet#283. [skip ci] Add musllinux wheels Specify the python needed. Seems to not be running the compiler? Guess the language. Building like normal failed to find source; try autobuild. Try adding CodeQL analysis. Add 3.11 to tox.ini and tests.yml ...
|
( CI finished, but it is only AppVeyor = Windows, on which the test is currently skipped due to lack of |
|
Thanks, @jamadden. For the reference: since greenlet 1 remains vulnerable to this bug, I've worked around it in https://lab.nexedi.com/kirr/pygolang/commit/4464662b. |
navytux
added a commit
to navytux/pygolang
that referenced
this pull request
Jan 26, 2022
…ia go Else as python-greenlet/greenlet#285 demonstrates there can be segmentation faults and crashes due to exceptions from one greenlet propagating to C stack of another greenlet. No test here. I've tried to do it, but with gevent (contrary to plain greenlets), spawning new task only schedules corresponding greenlet to run in the end of current event loop cycle instead of switching to created greenlet immediately. With this delaying, it was hard for me to develop corresponding test in a reasonable time. Hopefully having the test I've done for greenlet itself + hereby protection is good enough. /reviewed-on https://lab.nexedi.com/nexedi/pygolang/merge_requests/17
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello up there. While working on Pygolang I've faced the following
problem: a C++ exception thrown in one greenlet, without any try/catch
block on that greenlet, might become propagated and "handled" by
try/catch block established at another greenlet that happened to switch to
the greenlet in question. "Handled" comes in quotes because the program
usually segfaults after that.
I've also observed segfaults before exception reaches user-level
try/catch block - internally in __cxa_throw. Both this issues are likely
of the same origin - due to the fact that C-level stack of a greenlet
does not start from scratch and starts from C-level stack of the program
state when the greenlet is switched to the first time.
We already have one test for C++ exception handling in test_cpp that
@snaury initially added in d9cb12a. However in that test all greenlets
that throw C++ exceptions also put try/catch at the top of their
C-stack.
In the problem, that I describe, and that added test reproduces, there is
no intended top-level try/catch block in C-stack of the greenlet in
question that throws. As the test shows the exception becomes propagated
to switcher's greenlet context and the program then dies with SIGSEGV:
The C-level backtrace from the dumped core is attached in Appendix I. There the
program dies somehow after running the code from _test_extension_cpp.cpp
module. However with the following dirty-patch the situation becomes more clear:
The C-level backtrace of this abort when, run from under g2, shows that g2
C-stack starts from what was there before for first greenlet with try/catch block:
The problem might be worked-around with putting try/catch on C-stack for
every greenlet, but a more proper fix would be to unlink return address
and C-stack-frame of created greenlet from other greenlets completely
(i.e. rewrite first frame of C-stack for created greenlet to return to NULL).
Thanks beforehand,
Kirill
P.S. The problem described at
#197 (comment) and
#205 might be related
to hereby issue (/cc @ThePrez, @kadler, @jamadden).
P.P.S. The test does not fail on master. I see there @jamadden switched
the codebase to C++ almost completely and there is some explicit
attempts to save/restore exception state added in
87edf955. However
that exception save/restore is specific to Win32 and the test succeeds
for me on Linux. I still see the following comment in
UserGreenlet::inner_bootstrap
( https://github.com/python-greenlet/greenlet/blob/3e534d6b/src/greenlet/greenlet.cpp#L1085-L1104 )
probably this indeed works only by luck, or due to G_NOEXCEPT modifier
of UserGreenlet::inner_bootstrap upon seeing which __cxa_throw stops
unwinding C-stack and just calls std::terminate.
Appendix I. C-level backtrace after the test died with SIGSEGV