Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 2.7 test failures on several architectures #63

Open
gcsideal opened this issue Oct 9, 2014 · 38 comments
Open

Python 2.7 test failures on several architectures #63

gcsideal opened this issue Oct 9, 2014 · 38 comments

Comments

@gcsideal
Copy link
Contributor

gcsideal commented Oct 9, 2014

I try to compile 0.4.4 for various architectures. It compiles, but its Python 2.7 self-tests are failing with various code related errors like 'segmentation fault' and 'Illegal instruction'. You may see the last few lines of the tests at https://buildd.debian.org/status/package.php?p=python-greenlet&suite=unstable Can it be related to the ASM bits in greenlet? How can I help or give more information? I've some information that it works with Python 3.4, but I can't prove it (yet).

@snaury
Copy link
Contributor

snaury commented Oct 9, 2014

It almost certainly is due to something in asm bits, unfortunately I'm not very good with non-x86 architectures and cannot test them myself. If it is possible to get a shell with python and compilers on those architectures somewhere it would be best, if not then could you try building with CFLAGS=-save-temps and sending me resulting greenlet.s? I suspect there may be some optimization issue.

@gcsideal
Copy link
Contributor Author

I'll do my best, usually I neither have access to other architectures. On the other hand I can quote Matthias Klose from Ubuntu: "reproducible, no difference when building with -O0 or with gcc-4.8. The tests succeed for python3.4." Previously it was tried to build+test with gcc-4.9, thus it may not related to compiler and/or optimization but Python version.

@basak
Copy link

basak commented Oct 14, 2014

I've filed pull requests #64 and #65 which fixes armhf for me on Ubuntu Utopic. There were two issues, which together could result in -O0 not working around it.

I've also filed #66 which I think is a bigger issue. If other architectures are affected, this could be the reason.

@gcsideal
Copy link
Contributor Author

I see that you merged #64 and #65 . Do you intend to release a new version in the coming days or should I get a git snapshot packaged for Debian?

@snaury
Copy link
Contributor

snaury commented Oct 15, 2014

I will probably release new version this weekend.

@snaury
Copy link
Contributor

snaury commented Oct 18, 2014

@gcsideal just released 0.4.5, please enjoy!

@gcsideal
Copy link
Contributor Author

Thanks! Indeed, it fixes self-test build failures on armel, armhf and mipsel. Still fails on mips, powerpc, s390x and sparc. :( Please see: https://buildd.debian.org/status/package.php?p=python-greenlet&suite=unstable

@rafaelfolco
Copy link

rafaelfolco commented Mar 25, 2015

Upstream version of OpenStack on ppc64 platform is blocked by this issue.
A recent change on devstack dropped all rpm deps and started using pip versions. The result is a seg fault on "nova-manage db sync" during devstack deploy.

It seems to be an optimization issue. Building with -O1 works fine. The workaround is to force devstack installing greenlet-0.4.2 or 0.4.5 with -O1. Installing python-greenlet-0.4.2 from distro is also an alternative.

I assume the fix is in https://github.com/python-greenlet/greenlet/blob/master/platform/switch_ppc64_linux.h#L48 as described by #66.

Let me know what info I can provide or test I can run to help fixing this bug.

# export CFLAGS=-O1; ./setup.py install

running install
running build
running build_ext
building 'greenlet' extension
creating build
creating build/temp.linux-ppc64-2.7
gcc -pthread -fno-strict-aliasing -O3 -mtune=power7 -mcpu=power7 -g -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O3 -mtune=power7 -mcpu=power7 -g -D_GNU_SOURCE -fPIC -fwrapv -O1 -fPIC -I/usr/include/python2.7 -c greenlet.c -o build/temp.linux-ppc64-2.7/greenlet.o
creating build/lib.linux-ppc64-2.7
gcc -pthread -shared -Wl,-z,relro -O1 build/temp.linux-ppc64-2.7/greenlet.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-ppc64-2.7/greenlet.so
running install_lib
copying build/lib.linux-ppc64-2.7/greenlet.so -> /usr/lib64/python2.7/site-packages
running install_headers
running install_egg_info
Removing /usr/lib64/python2.7/site-packages/greenlet-0.4.5-py2.7.egg-info
Writing /usr/lib64/python2.7/site-packages/greenlet-0.4.5-py2.7.egg-info
# ./run-tests.py

running build_ext
Linking /tmp/greenlet/build/lib.linux-ppc64-2.7/greenlet.so to /tmp/greenlet/greenlet.so
python 2.7.8 (64 bit) using greenlet 0.4.5 from /tmp/greenlet/greenlet.so
test_dead_weakref (tests.test_weakref.WeakRefTests) ... ok
test_dealloc_weakref (tests.test_weakref.WeakRefTests) ... ok
test_inactive_weakref (tests.test_weakref.WeakRefTests) ... ok
test_version (tests.test_version.VersionTests) ... ok
test_exception_disables_tracing (tests.test_tracing.TracingTests) ... ok
test_greenlet_tracing (tests.test_tracing.TracingTests) ... ok
test_class (tests.test_throw.ThrowTests) ... ok
test_kill (tests.test_throw.ThrowTests) ... ok
test_throw_goes_to_original_parent (tests.test_throw.ThrowTests) ... ok
test_val (tests.test_throw.ThrowTests) ... ok
test_stack_saved (tests.test_stack_saved.Test) ... ok
test_arg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_kwarg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_adv_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_abstract_subclasses (tests.test_greenlet.GreenletTests) ... ok
test_dealloc (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_other_thread (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_switch_args_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_deepcopy (tests.test_greenlet.GreenletTests) ... ok
test_exc_state (tests.test_greenlet.GreenletTests) ... ok
test_exception (tests.test_greenlet.GreenletTests) ... ok
test_frame (tests.test_greenlet.GreenletTests) ... ok
test_implicit_parent_with_threads (tests.test_greenlet.GreenletTests) ... ok
test_instance_dict (tests.test_greenlet.GreenletTests) ... ok
test_parent_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_parent_restored_on_kill (tests.test_greenlet.GreenletTests) ... ok
test_parent_return_failure (tests.test_greenlet.GreenletTests) ... ok
test_recursive_startup (tests.test_greenlet.GreenletTests) ... ok
test_run_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_send_exception (tests.test_greenlet.GreenletTests) ... ok
test_simple (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs_to_parent (tests.test_greenlet.GreenletTests) ... ok
test_switch_to_another_thread (tests.test_greenlet.GreenletTests) ... ok
test_thread_bug (tests.test_greenlet.GreenletTests) ... ok
test_threaded_reparent (tests.test_greenlet.GreenletTests) ... ok
test_threaded_updatecurrent (tests.test_greenlet.GreenletTests) ... ok
test_threads (tests.test_greenlet.GreenletTests) ... ok
test_throw_doesnt_crash (tests.test_greenlet.GreenletTests) ... ok
test_throw_exception_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_tuple_subclass (tests.test_greenlet.GreenletTests) ... ok
test_two_children (tests.test_greenlet.GreenletTests) ... ok
test_two_recursive_children (tests.test_greenlet.GreenletTests) ... ok
test_unexpected_reparenting (tests.test_greenlet.GreenletTests) ... ok
test_genlet_bad (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_genlet_simple (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_layered_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_nested_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_permutations (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_generator (tests.test_generator.GeneratorTests) ... ok
test_circular_greenlet (tests.test_gc.GCTests) ... ok
test_dead_circular_ref (tests.test_gc.GCTests) ... ok
test_finalizer_crash (tests.test_gc.GCTests) ... ok
test_inactive_ref (tests.test_gc.GCTests) ... ok
test_getcurrent (tests.test_extension_interface.CAPITests) ... ok
test_new_greenlet (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_dead (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_error (tests.test_extension_interface.CAPITests) ... ok
test_setparent (tests.test_extension_interface.CAPITests) ... ok
test_switch (tests.test_extension_interface.CAPITests) ... ok
test_switch_kwargs (tests.test_extension_interface.CAPITests) ... ok
test_throw (tests.test_extension_interface.CAPITests) ... ok
test_exception_switch (tests.test_cpp.CPPTests) ... ok

---

Ran 64 tests in 0.398s

OK

@alexschm
Copy link
Contributor

Hi,

I submitted a pull request for this issue to fix s390x. While other archs are still broken, how about reintroducing the -fno-tree-dominator-opts CFLAG? The commit that removed the flag stated that this flag is obsolete and to be removed, but I could not find any evidence for this. It's still present in upstream gcc.

@snaury
Copy link
Contributor

snaury commented Apr 23, 2015

@alexschm that's odd, I rechecked and see now that it is indeed still supported even in gcc 5.1, and I can't remember where I saw that it was deprecated. Maybe I somehow confused gcc and clang together, where that flag was the most annoying: at first it was accepted, then it produced warnings, then it started to produce ugly errors. But to be honest I don't want to reintroduce that flag, it is a gcc-only band-aid, that won't work with anything else (even with gcc I'm not convinced it would always work, and not all platforms use gcc), it's always better to fix it properly (as you did for s390x).

@snaury
Copy link
Contributor

snaury commented Apr 23, 2015

@rafaelfolco I need a ppc64 instruction that can load a value 0 into a register (instead of return 0) to fix this, I'll see if I can find a cross-compiler, but it would be best if someone who knows the platform comes up with a fix.

@snaury
Copy link
Contributor

snaury commented Apr 23, 2015

@rafaelfolco ok, turns out ubuntu has a powerpc cross-compiler, so it was easy to find li does what I needed. Please try if HEAD now works with optimisations turned on.

@alexschm
Copy link
Contributor

@snaury I tested head on a ppc64 system and it works fine again, thanks!

@rafaelfolco
Copy link

@snaury I also tested HEAD on ppc64 with default optimization, and confirm that the issue has disappeared. Thanks for the fix !

@snaury
Copy link
Contributor

snaury commented May 3, 2015

@gcsideal just released 0.4.6, please see if the situation improved.

@gcsideal
Copy link
Contributor Author

gcsideal commented May 4, 2015

@snaury Debian supports fourteen architectures. The compile round of 0.4.6 revealed that the PowerPC and s390x issues are solved. SPARC is still failing:
copying build/lib.linux-sparc-2.7/greenlet.so -> build/test-2.7
PYTHONPATH=build/test-2.7 python2.7 run-tests.py -n
python 2.7.9 (32 bit) using greenlet 0.4.6 from /[BUILDDIR]/build/test-2.7/greenlet.so
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
test_getcurrent (tests.test_extension_interface.CAPITests) ... ok
test_new_greenlet (tests.test_extension_interface.CAPITests) ... Illegal instruction
make[1]: *** [test-2.7-stamp] Error 132

MIPS(EL) built with 0.4.5 but now fails with:
PYTHONPATH=build/test-3.4 python3.4 run-tests.py -n
python 3.4.3 (32 bit) using greenlet 0.4.6 from /[BUILDDIR]/build/test-3.4/greenlet.cpython-34m.so
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
test_dead_weakref (tests.test_weakref.WeakRefTests) ... ok
test_dealloc_weakref (tests.test_weakref.WeakRefTests) ... Exception ignored in: <greenlet.greenlet object at 0x77637cb0>
ok
test_inactive_weakref (tests.test_weakref.WeakRefTests) ... ok
test_version (tests.test_version.VersionTests) ... ok
test_exception_disables_tracing (tests.test_tracing.TracingTests) ... Illegal instruction
make[1]: *** [test-3.4-stamp] Error 132

As our build infrastructure is under load, I still wait for the result of PPC64 build. All other architectures could build 0.4.6 .

@snaury
Copy link
Contributor

snaury commented May 4, 2015

@gcsideal there have been no changes for mips between 0.4.5 and 0.4.6, but I notice now I overlooked a fix for return 0, so the failure was probably always there and by chance it started failing now. :-/

As for sparc, @grzn sent me #78 just a week ago and verified that sparc works.

@snaury
Copy link
Contributor

snaury commented May 4, 2015

@gcsideal could you please see if this patch would fix mips? snaury@fa4a72e

@gcsideal
Copy link
Contributor Author

gcsideal commented May 4, 2015

@snaury The MIPSel fix is correct. The changes from @grzn is in 0.4.6 right? Still fails on our SPARC machines. :( PPC64 build is working. This means only the SPARC issue needs some more work, the others are solved.

@grzn
Copy link
Contributor

grzn commented May 5, 2015

@snaury I meant that with that fix, greenlet does not segfault in the specifc use-case I had with Solaris-11.

I'll run the greenlet tests today and let you know.

@grzn
Copy link
Contributor

grzn commented May 5, 2015

so, on my solaris-11-sparc, the following test crashes the same way with v0.4.5, v0.4.5, and my fix:

test_exception_switch (tests.test_cpp.CPPTests) ... terminate called after throwing an instance of 'exception_t'
terminate called recursively

@gcsideal
Copy link
Contributor Author

gcsideal commented May 5, 2015

@grzn For me it's different and constant with previous versions, including v0.4.6:
python 2.7.9 (32 bit) using greenlet 0.4.6 from /[BUILDDIR]/build/test-2.7/greenlet.so
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
test_arg_refs (tests.test_leaks.ArgRefcountTests) ... Illegal instruction
make[1]: *** [test-2.7-stamp] Error 132

All other architectures are fine now.

@snaury
Copy link
Contributor

snaury commented May 10, 2015

I tried looking up on how sparc architecture works, and looking at the current code I have no clue how it was ever working, or how it is even supposed to work. If I'm reading it correctly ta 3 runs flushw in the kernel, which writes all saved register windows (except current) at their corresponding %sp places. There are multiple problems that I see:

  • The biggest problem on the save side is that %i7 that contains the return address is in the current window, so it is not spilled by flushw, and thus when SLP_SAVE_STATE tries to save the data it's probably some random garbage (fixing it would require doing flushw in slp_save_state, but it would only work if it allocates its own window, i.e. if compiler doesn't decide to omit save and restore for some reason, however since it's not a leaf function I'd guess such optimization is unlikely). However I can't find whether ta 3 (which executes flushw in the kernel) allocates a new register window, so perhaps just because it's done with ta 3 instead of flushw the registers actually happen to be spilled by chance.
  • The biggest problem on the restore side is that even if we restore the stack as it was, it doesn't change %i7, so the return address is likely wrong and would return incorrectly.
  • Another problem is that ST_CLEAN_WINDOWS seems to be about wiping registers of data from other threads, but as far as I can see in the Linux kernel source it's actually a bad trap and cannot be used. Besides, I don't see how someone would need to do that anyway, since context switching shouldn't change anything about windows in the current task.
  • I'm thinking that to properly restore the state of slp_switch itself (and thus of %i7 among other things) it should do a ta 3 at the start of slp_restore_state. This way stack will be spilled into some random location, but we don't care about that, since all we need is registers restored after we overwrite the stack.

Given all that I have a preliminary fix in https://github.com/snaury/greenlet/tree/fix-sparc (@grzn, @gcsideal, please test snaury@081fd2e if possible, I would clean it up later by moving sparc-specific bits out of greenlet.c, but I need to know if the idea works)

@grzn
Copy link
Contributor

grzn commented May 10, 2015

still not working:

root@host-vsparc01 (SunOS 5.11) ➜  greenlet-fix-sparc  python run-tests.py                                  15-05-10 09:02
running build_ext
Linking /root/greenlet-fix-sparc/build/lib.solaris-2.11-sun4v.64bit-2.7/greenlet.so to /root/greenlet-fix-sparc/greenlet.so
python 2.7.8 (64 bit) using greenlet 0.4.6 from /root/greenlet-fix-sparc/greenlet.so
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
test_dead_weakref (tests.test_weakref.WeakRefTests) ... ok
test_dealloc_weakref (tests.test_weakref.WeakRefTests) ... ok
test_inactive_weakref (tests.test_weakref.WeakRefTests) ... ok
test_circular_greenlet (tests.test_gc.GCTests) ... ok
test_dead_circular_ref (tests.test_gc.GCTests) ... ok
test_finalizer_crash (tests.test_gc.GCTests) ... ok
test_inactive_ref (tests.test_gc.GCTests) ... ok
test_genlet_bad (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_genlet_simple (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_layered_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_nested_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_permutations (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_abstract_subclasses (tests.test_greenlet.GreenletTests) ... ok
test_dealloc (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_other_thread (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_switch_args_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_deepcopy (tests.test_greenlet.GreenletTests) ... ok
test_exc_state (tests.test_greenlet.GreenletTests) ... ok
test_exception (tests.test_greenlet.GreenletTests) ... ok
test_frame (tests.test_greenlet.GreenletTests) ... ok
test_implicit_parent_with_threads (tests.test_greenlet.GreenletTests) ... ok
test_instance_dict (tests.test_greenlet.GreenletTests) ... ok
test_parent_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_parent_restored_on_kill (tests.test_greenlet.GreenletTests) ... ok
test_parent_return_failure (tests.test_greenlet.GreenletTests) ... ok
test_recursive_startup (tests.test_greenlet.GreenletTests) ... ok
test_run_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_send_exception (tests.test_greenlet.GreenletTests) ... ok
test_simple (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs_to_parent (tests.test_greenlet.GreenletTests) ... ok
test_switch_to_another_thread (tests.test_greenlet.GreenletTests) ... ok
test_thread_bug (tests.test_greenlet.GreenletTests) ... ok
test_threaded_reparent (tests.test_greenlet.GreenletTests) ... ok
test_threaded_updatecurrent (tests.test_greenlet.GreenletTests) ... ok
test_threads (tests.test_greenlet.GreenletTests) ... ok
test_throw_doesnt_crash (tests.test_greenlet.GreenletTests) ... ok
test_throw_exception_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_tuple_subclass (tests.test_greenlet.GreenletTests) ... ok
test_two_children (tests.test_greenlet.GreenletTests) ... ok
test_two_recursive_children (tests.test_greenlet.GreenletTests) ... ok
test_unexpected_reparenting (tests.test_greenlet.GreenletTests) ... ok
test_arg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_kwarg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_adv_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_class (tests.test_throw.ThrowTests) ... ok
test_kill (tests.test_throw.ThrowTests) ... ok
test_throw_goes_to_original_parent (tests.test_throw.ThrowTests) ... ok
test_val (tests.test_throw.ThrowTests) ... ok
test_getcurrent (tests.test_extension_interface.CAPITests) ... ok
test_new_greenlet (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_dead (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_error (tests.test_extension_interface.CAPITests) ... ok
test_setparent (tests.test_extension_interface.CAPITests) ... ok
test_switch (tests.test_extension_interface.CAPITests) ... ok
test_switch_kwargs (tests.test_extension_interface.CAPITests) ... ok
test_throw (tests.test_extension_interface.CAPITests) ... ok
test_exception_disables_tracing (tests.test_tracing.TracingTests) ... ok
test_greenlet_tracing (tests.test_tracing.TracingTests) ... ok
test_generator (tests.test_generator.GeneratorTests) ... ok
test_exception_switch (tests.test_cpp.CPPTests) ... terminate called after throwing an instance of 'exception_t'
terminate called recursively

@snaury
Copy link
Contributor

snaury commented May 10, 2015

@grzn C++ tests only work when exceptions don't use global variables (e.g. gcc with dwarf exceptions), if your compiler uses sjlj-style exceptions you should set GREENLET_TEST_CPP=no in your environment, since it cannot be supported.

@grzn
Copy link
Contributor

grzn commented May 10, 2015

Okay with that is works:

root@host-vsparc06 (SunOS 5.11) ➜  greenlet-fix-sparc  GREENLET_TEST_CPP=no python run-tests.py                                                               15-05-10 11:23
running build_ext
Linking /root/greenlet-fix-sparc/build/lib.solaris-2.11-sun4v.64bit-2.7/greenlet.so to /root/greenlet-fix-sparc/greenlet.so
python 2.7.8 (64 bit) using greenlet 0.4.6 from /root/greenlet-fix-sparc/greenlet.so
test_dead_weakref (tests.test_weakref.WeakRefTests) ... ok
test_dealloc_weakref (tests.test_weakref.WeakRefTests) ... ok
test_inactive_weakref (tests.test_weakref.WeakRefTests) ... ok
test_class (tests.test_throw.ThrowTests) ... ok
test_kill (tests.test_throw.ThrowTests) ... ok
test_throw_goes_to_original_parent (tests.test_throw.ThrowTests) ... ok
test_val (tests.test_throw.ThrowTests) ... ok
test_arg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_kwarg_refs (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_adv_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_threaded_leak (tests.test_leaks.ArgRefcountTests) ... ok
test_exception_disables_tracing (tests.test_tracing.TracingTests) ... ok
test_greenlet_tracing (tests.test_tracing.TracingTests) ... ok
test_generator (tests.test_generator.GeneratorTests) ... ok
test_stack_saved (tests.test_stack_saved.Test) ... ok
test_genlet_bad (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_genlet_simple (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_layered_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_nested_genlets (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_permutations (tests.test_generator_nested.NestedGeneratorTests) ... ok
test_version (tests.test_version.VersionTests) ... ok
test_getcurrent (tests.test_extension_interface.CAPITests) ... ok
test_new_greenlet (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_dead (tests.test_extension_interface.CAPITests) ... ok
test_raise_greenlet_error (tests.test_extension_interface.CAPITests) ... ok
test_setparent (tests.test_extension_interface.CAPITests) ... ok
test_switch (tests.test_extension_interface.CAPITests) ... ok
test_switch_kwargs (tests.test_extension_interface.CAPITests) ... ok
test_throw (tests.test_extension_interface.CAPITests) ... ok
test_abstract_subclasses (tests.test_greenlet.GreenletTests) ... ok
test_dealloc (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_other_thread (tests.test_greenlet.GreenletTests) ... ok
test_dealloc_switch_args_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_deepcopy (tests.test_greenlet.GreenletTests) ... ok
test_exc_state (tests.test_greenlet.GreenletTests) ... ok
test_exception (tests.test_greenlet.GreenletTests) ... ok
test_frame (tests.test_greenlet.GreenletTests) ... ok
test_implicit_parent_with_threads (tests.test_greenlet.GreenletTests) ... ok
test_instance_dict (tests.test_greenlet.GreenletTests) ... ok
test_parent_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_parent_restored_on_kill (tests.test_greenlet.GreenletTests) ... ok
test_parent_return_failure (tests.test_greenlet.GreenletTests) ... ok
test_recursive_startup (tests.test_greenlet.GreenletTests) ... ok
test_run_equals_None (tests.test_greenlet.GreenletTests) ... ok
test_send_exception (tests.test_greenlet.GreenletTests) ... ok
test_simple (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs (tests.test_greenlet.GreenletTests) ... ok
test_switch_kwargs_to_parent (tests.test_greenlet.GreenletTests) ... ok
test_switch_to_another_thread (tests.test_greenlet.GreenletTests) ... ok
test_thread_bug (tests.test_greenlet.GreenletTests) ... ok
test_threaded_reparent (tests.test_greenlet.GreenletTests) ... ok
test_threaded_updatecurrent (tests.test_greenlet.GreenletTests) ... ok
test_threads (tests.test_greenlet.GreenletTests) ... ok
test_throw_doesnt_crash (tests.test_greenlet.GreenletTests) ... ok
test_throw_exception_not_lost (tests.test_greenlet.GreenletTests) ... ok
test_tuple_subclass (tests.test_greenlet.GreenletTests) ... ok
test_two_children (tests.test_greenlet.GreenletTests) ... ok
test_two_recursive_children (tests.test_greenlet.GreenletTests) ... ok
test_unexpected_reparenting (tests.test_greenlet.GreenletTests) ... ok
test_circular_greenlet (tests.test_gc.GCTests) ... ok
test_dead_circular_ref (tests.test_gc.GCTests) ... ok
test_finalizer_crash (tests.test_gc.GCTests) ... ok
test_inactive_ref (tests.test_gc.GCTests) ... ok

----------------------------------------------------------------------
Ran 63 tests in 0.762s

OK

but 0.4.6 also works with this environment variable...

@snaury
Copy link
Contributor

snaury commented May 10, 2015

@grzn what if you remove os.environ["CFLAGS"] = ... at the top of setup.py, would it still work in both cases? I suspect -Os is doing something that is hiding the problem, and I hope new code doesn't need it.

@grzn
Copy link
Contributor

grzn commented May 10, 2015

greenlet-0.4.6 passes with -O3 and GREENLET_TEST_CPP=no:

gcc -fno-strict-aliasing -I/root/python/include -m64 -I/root/python/include/gnutls -I/root/python/include/graphviz -I/root/python/include/libexslt -I/root/python/include/libxml2 -I/root/python/include/libxslt -I/root/python/include/ncurses -I/root/python/include/openssl -I/root/python/include/readline -I/root/python/include/sasl -I/root/python/lib64/libffi-3.2.1/include -DNDEBUG -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -m64 -fPIC -I/root/python/include/python2.7 -c greenlet.c -o build/temp.solaris-2.11-sun4v.64bit-2.7/greenlet.o

and fails without the GREENLET_TEST_CPP=no

@snaury
Copy link
Contributor

snaury commented May 10, 2015

@grzn ok, thanks, I tried looking into illumos and found one implementation of ST_CLEAN_WINDOWS that does something like save; flushw; restore, which would (almost) explain how it worked on SunOS before. I hope new patch fixes Debian as well.

@grzn
Copy link
Contributor

grzn commented May 10, 2015

Alright; let me know when to run your patch through the scenario i have that caused 0.4.5 to crash with segmentation fault.


Sent from Mailbox

On Sun, May 10, 2015 at 4:34 PM, Alexey Borzenkov
notifications@github.com wrote:

@grzn ok, thanks, I tried looking into illumos and found one implementation of ST_CLEAN_WINDOWS that does something like save; flushw; restore, which would (almost) explain how it worked on SunOS before. I hope new patch fixes Debian as well.

Reply to this email directly or view it on GitHub:
#63 (comment)

@snaury
Copy link
Contributor

snaury commented May 16, 2015

@gcsideal I haven't heard from you, but I had to release 0.4.7 for unrelated reasons (python 2.5 compatibility), so I included new sparc trick with it as well. Hope if finally works.

@gcsideal
Copy link
Contributor Author

@snaury I couldn't test it (I can't reach a Sparc machine directly), so I didn't write. Now 0.4.7 is out for testing, will write about how it goes.

@gcsideal
Copy link
Contributor Author

@snaury Now greenlet built on Sparc and on our other platforms as well, except kFreeBSD ones. The latter just need more time and as it was OK in the past I think those will be fine.

@rafaelfolco
Copy link

@snaury, greenlet is producing seg fault on ppc64le arch, which has the same registers as ppc64, but it is powerpc64le arch with different ABI. It apparently selects the right include file:platform/switch_ppc64_linux.h.
When I add a printf("xx") with 2+ chars it works fine. The seg fault vanishes.
Could you please give any sugegstions on how to fix this ? Thanks.

Linking /root/greenlet/build/lib.linux-ppc64le-2.7/greenlet.so to /root/greenlet/greenlet.so
python 2.7.8 (64 bit) using greenlet 0.4.7 from /root/greenlet/greenlet.so
test_dead_weakref (tests.test_weakref.WeakRefTests) ... Segmentation fault (core dumped)

@snaury
Copy link
Contributor

snaury commented Aug 3, 2015

@rafaelfolco see #89 for support, I'm waiting for an #ifdef in that PR before merging.

@awilfox
Copy link

awilfox commented Oct 1, 2015

I am not sure if this is a regression, but on greenlet 0.4.9 + gcc-4.8.4 + Python 3.4 + Linux 3.18 on ppc32 (big endian, PowerPC G4), I am hitting the same issue as @rafaelfolco pointed out in March, and the -O1 addition to CFLAGS has fixed it for me (all tests pass and the server runs great).

I'm afraid the code is all a little too "magick" for me to understand properly to suggest a fix, but I can provide core dumps and generated assembly if necessary.

@snaury
Copy link
Contributor

snaury commented Oct 1, 2015

@awilfox yes, generated assembly would be very useful, could you please post a gist of the assembly that causes crashes and the assembly that doesn't?

@snaury
Copy link
Contributor

snaury commented Oct 1, 2015

@awilfox I created #94 for the issue, we should continue discussion there.

navytux added a commit to navytux/greenlet that referenced this issue Jan 10, 2022
…opagate to another greenlet and corrupt memory

Hello up there. While working on Pygolang I've faced the following
problem: a C++ exception thrown in one greenlet, without any try/catch
block on that greenlet, might become propagated and "handled" by
try/catch block established at another greenlet that happened to switch to
the greenlet in question. "Handled" comes in quotes because the program
usually segfaults after that.

I've also observed segfaults before exception reaches user-level
try/catch block - internally in __cxa_throw. Both this issues are likely
of the same origin - due to the fact that C-level stack of a greenlet
does not start from scratch and starts from C-level stack of the program
state when the greenlet is switched to the first time.

We already have one test for C++ exception handling in test_cpp that
@snaury initially added in d9cb12a. However in that test all greenlets
that throw C++ exceptions also put try/catch at the top of their
C-stack.

In the problem, that I describe, and that added test reproduces, there is
no intended top-level try/catch block in C-stack of the greenlet in
question that throws. As the test shows the exception becomes propagated
to switcher's greenlet context and the program then dies with SIGSEGV:

    test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests) ... terminate called after throwing an instance of 'exception_t'
    C++ exception unexpectedly caught in g1		<-- NOTE
    FAIL

    ======================================================================
    FAIL: test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/kirr/src/tools/py/gevent/greenlet/src/greenlet/tests/test_cpp.py", line 62, in test_exception_switch_and_throw
        (ret, sig, " (core dumped)" if core else ""))
    AssertionError: failed with ret=0 sig=11 (core dumped)

The C-level backtrace from the dumped core is attached in Appendix I. There the
program dies somehow after running the code from _test_extension_cpp.cpp
module. However with the following dirty-patch the situation becomes more clear:

    --- a/src/greenlet/tests/_test_extension_cpp.cpp
    +++ b/src/greenlet/tests/_test_extension_cpp.cpp
    @@ -70,6 +70,7 @@ test_exception_switch(PyObject* self, PyObject* args)
     static PyObject*
     py_test_exception_throw(PyObject* self, PyObject* args)
     {
    +    abort();
         if (!PyArg_ParseTuple(args, ""))
             return NULL;
         p_test_exception_throw(0);

The C-level backtrace of this abort when, run from under g2, shows that g2
C-stack starts from what was there before for first greenlet with try/catch block:

    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
    python-greenlet#1  0x00007f8be31e9537 in __GI_abort () at abort.c:79
    python-greenlet#2  0x00007f8be3529423 in py_test_exception_throw (self=0x0, args=0x7f8be317f050)
        at src/greenlet/tests/_test_extension_cpp.cpp:73
    python-greenlet#3  0x00005584d0389903 in PyObject_Call (func=0x7f8be2cdd690, arg=<optimized out>, kw=<optimized out>)
        at ../Objects/abstract.c:2544
    python-greenlet#4  0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b1c88) at src/greenlet/greenlet.c:931
    python-greenlet#5  0x00007f8be3530290 in g_switch (target=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692
    python-greenlet#6  0x00007f8be35329e2 in PyGreenlet_Switch (g=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1806     <- first switch to G2
    python-greenlet#7  0x00007f8be35294da in test_exception_switch_and_do_in_g2 (self=0x0, args=0x7f8be2ce33d0)                                        <- this code runs in G1
        at src/greenlet/tests/_test_extension_cpp.cpp:105
    python-greenlet#8  0x00005584d039de7a in call_function (oparg=<optimized out>, pp_stack=0x7ffe6d1b1e28) at ../Python/ceval.c:4376
    python-greenlet#9  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
    python-greenlet#10 0x00005584d039c3cc in PyEval_EvalCodeEx (co=0x7f8be2cd9cb0, globals=<optimized out>, locals=<optimized out>,
        args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
        at ../Python/ceval.c:3608
    python-greenlet#11 0x00005584d03b6b9b in function_call (func=func@entry=0x7f8be2cde650, arg=0x7f8be317f050, kw=0x0)
        at ../Objects/funcobject.c:523
    python-greenlet#12 0x00005584d0389903 in PyObject_Call (func=0x7f8be2cde650, arg=<optimized out>, kw=<optimized out>)
        at ../Objects/abstract.c:2544
    python-greenlet#13 0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b20b8) at src/greenlet/greenlet.c:931
    python-greenlet#14 0x00007f8be3530290 in g_switch (target=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692
    python-greenlet#15 0x00007f8be3531c4c in green_switch (self=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1321       <- switch to G1
    ...

The problem might be worked-around with putting try/catch on C-stack for
every greenlet, but a more proper fix would be to unlink return address
and C-stack-frame of created greenlet from other greenlets completely
(i.e. rewrite first frame of C-stack for created greenlet to return to NULL).

Thanks beforehand,
Kirill

P.S. The problem described at
  python-greenlet#197 (comment) and
  python-greenlet#205 might be related
  to hereby issue (/cc @ThePrez, @kadler, @jamadden).

P.P.S. The test does not fail on master. I see there @jamadden switched
  the codebase to C++ almost completely and there is some explicit
  attempts to save/restore exception state added in
  python-greenlet@87edf955. However
  that exception save/restore is specific to Win32 and the test succeeds
  for me on Linux. I still see the following comment in
  UserGreenlet::inner_bootstrap

    // C++ exceptions cannot propagate to the parent greenlet from
    // here. (TODO: Do we need a catch(...) clause, perhaps on the
    // function itself? ALl we could do is terminate the program.)

    ( https://github.com/python-greenlet/greenlet/blob/3e534d6b/src/greenlet/greenlet.cpp#L1085-L1104 )

  probably this indeed works only by luck, or due to G_NOEXCEPT modifier
  of UserGreenlet::inner_bootstrap upon seeing which __cxa_throw stops
  unwinding C-stack and just calls std::terminate.

--------

Appendix I. C-level backtrace after the test died with SIGSEGV

  #0  0x0000556bdc30d394 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3351
  python-greenlet#1  0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03acb0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#2  0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a03f650, arg=0x7f0b8a4e0050, kw=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#3  0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03f650, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#4  0x00007f0b8a891d63 in g_initialstub (mark=0x7ffc3ef97988) at src/greenlet/greenlet.c:931
  python-greenlet#5  0x00007f0b8a891290 in g_switch (target=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:692
  python-greenlet#6  0x00007f0b8a892c4c in green_switch (self=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:1321
  python-greenlet#7  0x0000556bdc30ce7a in call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ac8) at ../Python/ceval.c:4376
  python-greenlet#8  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#9  0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ad30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#10 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97ca8,
      func=0x7f0b8a03aed0) at ../Python/ceval.c:4471
  python-greenlet#11 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ca8) at ../Python/ceval.c:4396
  python-greenlet#12 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#13 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ac30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#14 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97e88,
      func=0x7f0b8a03af50) at ../Python/ceval.c:4471
  python-greenlet#15 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97e88) at ../Python/ceval.c:4396
  python-greenlet#16 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#17 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03adb0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#18 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef98068,
      func=0x7f0b8a03f250) at ../Python/ceval.c:4471
  python-greenlet#19 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98068) at ../Python/ceval.c:4396
  python-greenlet#20 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#21 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bd30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a0752a8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#22 0x0000556bdc325cc2 in function_call (func=0x7f0b8a0771d0, arg=0x7f0b8a3bd5a0, kw=0x7f0b8a042dd0)
      at ../Objects/funcobject.c:523
  python-greenlet#23 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042dd0, arg=0x7f0b8a3bd5a0, func=0x7f0b8a0771d0)
      at ../Objects/abstract.c:2544
  python-greenlet#24 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef982a0, func=0x7f0b8a0771d0)
      at ../Python/ceval.c:4690
  python-greenlet#25 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  python-greenlet#26 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bf30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#27 0x0000556bdc325b9b in function_call (func=0x7f0b8a0772d0, arg=arg@entry=0x7f0b8a3bd550, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#28 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd550, func=<optimized out>) at ../Objects/abstract.c:2544
  python-greenlet#29 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6a00, arg=0x7f0b8a3bd550, kw=0x0)
      at ../Objects/classobject.c:2600
  python-greenlet#30 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6a00, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#31 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cfd0, args=0x7f0b8a044210, kwds=0x0)
      at ../Objects/typeobject.c:5609
  python-greenlet#32 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cfd0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#33 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98798, func=0x7f0b8a03cfd0)
      at ../Python/ceval.c:4593
  python-greenlet#34 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98798) at ../Python/ceval.c:4398
  python-greenlet#35 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#36 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#37 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd780, kw=0x7f0b8a042cb0)
      at ../Objects/funcobject.c:523
  python-greenlet#38 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042cb0, arg=0x7f0b8a3bd780, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  python-greenlet#39 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef989d0, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  python-greenlet#40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  python-greenlet#41 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#42 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd730, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#43 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd730, func=<optimized out>) at ../Objects/abstract.c:2544
  python-greenlet#44 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427190, arg=0x7f0b8a3bd730, kw=0x0)
      at ../Objects/classobject.c:2600
  python-greenlet#45 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a427190, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#46 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cf90, args=0x7f0b8a044150, kwds=0x0)
      at ../Objects/typeobject.c:5609
  python-greenlet#47 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cf90, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#48 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98ec8, func=0x7f0b8a03cf90)
      at ../Python/ceval.c:4593
  python-greenlet#49 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98ec8) at ../Python/ceval.c:4398
  python-greenlet#50 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#51 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#52 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bdaf0, kw=0x7f0b8a042b90)
      at ../Objects/funcobject.c:523
  python-greenlet#53 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042b90, arg=0x7f0b8a3bdaf0, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  python-greenlet#54 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99100, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  python-greenlet#55 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  python-greenlet#56 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#57 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bdb40, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#58 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bdb40, func=<optimized out>) at ../Objects/abstract.c:2544
  python-greenlet#59 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6820, arg=0x7f0b8a3bdb40, kw=0x0)
      at ../Objects/classobject.c:2600
  python-greenlet#60 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6820, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#61 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c50, args=0x7f0b8a03c0d0, kwds=0x0)
      at ../Objects/typeobject.c:5609
  python-greenlet#62 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c50, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#63 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef995f8, func=0x7f0b8a033c50)
      at ../Python/ceval.c:4593
  python-greenlet#64 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef995f8) at ../Python/ceval.c:4398
  python-greenlet#65 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#66 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#67 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd500, kw=0x7f0b8a042a70)
      at ../Objects/funcobject.c:523
  python-greenlet#68 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042a70, arg=0x7f0b8a3bd500, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  python-greenlet#69 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99830, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  python-greenlet#70 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  python-greenlet#71 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#72 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd9b0, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#73 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd9b0, func=<optimized out>) at ../Objects/abstract.c:2544
  python-greenlet#74 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a4310a0, arg=0x7f0b8a3bd9b0, kw=0x0)
      at ../Objects/classobject.c:2600
  python-greenlet#75 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a4310a0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#76 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c10, args=0x7f0b8a033fd0, kwds=0x0)
      at ../Objects/typeobject.c:5609
  python-greenlet#77 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c10, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#78 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef99d28, func=0x7f0b8a033c10)
      at ../Python/ceval.c:4593
  python-greenlet#79 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99d28) at ../Python/ceval.c:4398
  python-greenlet#80 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#81 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99e38,
      func=0x7f0b8a01ecd0) at ../Python/ceval.c:4461
  python-greenlet#82 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99e38) at ../Python/ceval.c:4396
  python-greenlet#83 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#84 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99f48,
      func=0x7f0b8a01edd0) at ../Python/ceval.c:4461
  python-greenlet#85 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99f48) at ../Python/ceval.c:4396
  python-greenlet#86 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#87 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a081d30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=1, defs=0x7f0b8a008ba8, defcount=10,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#88 0x0000556bdc325cc2 in function_call (func=0x7f0b8a00ddd0, arg=arg@entry=0x7f0b8a033610, kw=kw@entry=0x7f0b8a014a70)
      at ../Objects/funcobject.c:523
  python-greenlet#89 0x0000556bdc33f553 in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a033610, func=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#90 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427140, arg=0x7f0b8a033610, arg@entry=0x7f0b8a4e0050,
      kw=kw@entry=0x7f0b8a014a70) at ../Objects/classobject.c:2600
  python-greenlet#91 0x0000556bdc33f10f in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a4e0050, func=0x7f0b8a427140)
      at ../Objects/abstract.c:2544
  python-greenlet#92 slot_tp_init (self=self@entry=0x7f0b8a3ebb10, args=args@entry=0x7f0b8a4e0050, kwds=kwds@entry=0x7f0b8a014a70)
      at ../Objects/typeobject.c:5869
  python-greenlet#93 0x0000556bdc2feb87 in type_call (type=<optimized out>, type@entry=0x556bdd2a85c0, args=0x7f0b8a4e0050, kwds=0x7f0b8a014a70)
      at ../Objects/typeobject.c:765
  python-greenlet#94 0x0000556bdc2f8903 in PyObject_Call (func=0x556bdd2a85c0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#95 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef9a458, func=0x556bdd2a85c0)
      at ../Python/ceval.c:4593
  python-greenlet#96 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a458) at ../Python/ceval.c:4398
  python-greenlet#97 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#98 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d830, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  python-greenlet#99 0x0000556bdc311886 in PyEval_EvalCode (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, co=<optimized out>)
      at ../Python/ceval.c:669
  python-greenlet#100 exec_statement (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, prog=<optimized out>, f=<optimized out>)
      at ../Python/ceval.c:5093
  python-greenlet#101 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:2122
  python-greenlet#102 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a3ea4b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a381068, defcount=5,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#103 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef9a818,
      func=0x7f0b8a37d8d0) at ../Python/ceval.c:4471
  python-greenlet#104 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a818) at ../Python/ceval.c:4396
  python-greenlet#105 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  python-greenlet#106 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d4b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a379ce8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  python-greenlet#107 0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a37de50, arg=0x7f0b8a3b5be0, kw=0x0)
      at ../Objects/funcobject.c:523
  python-greenlet#108 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a37de50, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  python-greenlet#109 0x0000556bdc3b87e1 in RunModule (module=<optimized out>, set_argv0=1) at ../Modules/main.c:197
  python-greenlet#110 0x0000556bdc3a68ed in Py_Main (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:592#111 0x00007f0b8a54bd0a in __libc_start_main (main=0x556bdc3a6530 <main>, argc=5, argv=0x7ffc3ef9ac08, init=<optimized out>,
      fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc3ef9abf8) at ../csu/libc-start.c:308
  python-greenlet#112 0x0000556bdc3a646a in _start ()
jamadden pushed a commit that referenced this issue Jan 24, 2022
…opagate to another greenlet and corrupt memory

Hello up there. While working on Pygolang I've faced the following
problem: a C++ exception thrown in one greenlet, without any try/catch
block on that greenlet, might become propagated and "handled" by
try/catch block established at another greenlet that happened to switch to
the greenlet in question. "Handled" comes in quotes because the program
usually segfaults after that.

I've also observed segfaults before exception reaches user-level
try/catch block - internally in __cxa_throw. Both this issues are likely
of the same origin - due to the fact that C-level stack of a greenlet
does not start from scratch and starts from C-level stack of the program
state when the greenlet is switched to the first time.

We already have one test for C++ exception handling in test_cpp that
@snaury initially added in d9cb12a. However in that test all greenlets
that throw C++ exceptions also put try/catch at the top of their
C-stack.

In the problem, that I describe, and that added test reproduces, there is
no intended top-level try/catch block in C-stack of the greenlet in
question that throws. As the test shows the exception becomes propagated
to switcher's greenlet context and the program then dies with SIGSEGV:

    test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests) ... terminate called after throwing an instance of 'exception_t'
    C++ exception unexpectedly caught in g1		<-- NOTE
    FAIL

    ======================================================================
    FAIL: test_exception_switch_and_throw (greenlet.tests.test_cpp.CPPTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/kirr/src/tools/py/gevent/greenlet/src/greenlet/tests/test_cpp.py", line 62, in test_exception_switch_and_throw
        (ret, sig, " (core dumped)" if core else ""))
    AssertionError: failed with ret=0 sig=11 (core dumped)

The C-level backtrace from the dumped core is attached in Appendix I. There the
program dies somehow after running the code from _test_extension_cpp.cpp
module. However with the following dirty-patch the situation becomes more clear:

    --- a/src/greenlet/tests/_test_extension_cpp.cpp
    +++ b/src/greenlet/tests/_test_extension_cpp.cpp
    @@ -70,6 +70,7 @@ test_exception_switch(PyObject* self, PyObject* args)
     static PyObject*
     py_test_exception_throw(PyObject* self, PyObject* args)
     {
    +    abort();
         if (!PyArg_ParseTuple(args, ""))
             return NULL;
         p_test_exception_throw(0);

The C-level backtrace of this abort when, run from under g2, shows that g2
C-stack starts from what was there before for first greenlet with try/catch block:

    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
    #1  0x00007f8be31e9537 in __GI_abort () at abort.c:79
    #2  0x00007f8be3529423 in py_test_exception_throw (self=0x0, args=0x7f8be317f050)
        at src/greenlet/tests/_test_extension_cpp.cpp:73
    #3  0x00005584d0389903 in PyObject_Call (func=0x7f8be2cdd690, arg=<optimized out>, kw=<optimized out>)
        at ../Objects/abstract.c:2544
    #4  0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b1c88) at src/greenlet/greenlet.c:931
    #5  0x00007f8be3530290 in g_switch (target=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692
    #6  0x00007f8be35329e2 in PyGreenlet_Switch (g=0x7f8be2cd6230, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1806     <- first switch to G2
    #7  0x00007f8be35294da in test_exception_switch_and_do_in_g2 (self=0x0, args=0x7f8be2ce33d0)                                        <- this code runs in G1
        at src/greenlet/tests/_test_extension_cpp.cpp:105
    #8  0x00005584d039de7a in call_function (oparg=<optimized out>, pp_stack=0x7ffe6d1b1e28) at ../Python/ceval.c:4376
    #9  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
    #10 0x00005584d039c3cc in PyEval_EvalCodeEx (co=0x7f8be2cd9cb0, globals=<optimized out>, locals=<optimized out>,
        args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
        at ../Python/ceval.c:3608
    #11 0x00005584d03b6b9b in function_call (func=func@entry=0x7f8be2cde650, arg=0x7f8be317f050, kw=0x0)
        at ../Objects/funcobject.c:523
    #12 0x00005584d0389903 in PyObject_Call (func=0x7f8be2cde650, arg=<optimized out>, kw=<optimized out>)
        at ../Objects/abstract.c:2544
    #13 0x00007f8be3530d63 in g_initialstub (mark=0x7ffe6d1b20b8) at src/greenlet/greenlet.c:931
    #14 0x00007f8be3530290 in g_switch (target=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:692
    #15 0x00007f8be3531c4c in green_switch (self=0x7f8be2cd6410, args=0x7f8be317f050, kwargs=0x0) at src/greenlet/greenlet.c:1321       <- switch to G1
    ...

The problem might be worked-around with putting try/catch on C-stack for
every greenlet, but a more proper fix would be to unlink return address
and C-stack-frame of created greenlet from other greenlets completely
(i.e. rewrite first frame of C-stack for created greenlet to return to NULL).

Thanks beforehand,
Kirill

P.S. The problem described at
  #197 (comment) and
  #205 might be related
  to hereby issue (/cc @ThePrez, @kadler, @jamadden).

P.P.S. The test does not fail on master. I see there @jamadden switched
  the codebase to C++ almost completely and there is some explicit
  attempts to save/restore exception state added in
  87edf955. However
  that exception save/restore is specific to Win32 and the test succeeds
  for me on Linux. I still see the following comment in
  UserGreenlet::inner_bootstrap

    // C++ exceptions cannot propagate to the parent greenlet from
    // here. (TODO: Do we need a catch(...) clause, perhaps on the
    // function itself? ALl we could do is terminate the program.)

    ( https://github.com/python-greenlet/greenlet/blob/3e534d6b/src/greenlet/greenlet.cpp#L1085-L1104 )

  probably this indeed works only by luck, or due to G_NOEXCEPT modifier
  of UserGreenlet::inner_bootstrap upon seeing which __cxa_throw stops
  unwinding C-stack and just calls std::terminate.

--------

Appendix I. C-level backtrace after the test died with SIGSEGV

  #0  0x0000556bdc30d394 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3351
  #1  0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03acb0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #2  0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a03f650, arg=0x7f0b8a4e0050, kw=0x0)
      at ../Objects/funcobject.c:523
  #3  0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03f650, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #4  0x00007f0b8a891d63 in g_initialstub (mark=0x7ffc3ef97988) at src/greenlet/greenlet.c:931
  #5  0x00007f0b8a891290 in g_switch (target=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:692
  #6  0x00007f0b8a892c4c in green_switch (self=0x7f0b8a037410, args=0x7f0b8a4e0050, kwargs=0x0) at src/greenlet/greenlet.c:1321
  #7  0x0000556bdc30ce7a in call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ac8) at ../Python/ceval.c:4376
  #8  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #9  0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ad30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #10 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97ca8,
      func=0x7f0b8a03aed0) at ../Python/ceval.c:4471
  #11 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97ca8) at ../Python/ceval.c:4396
  #12 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #13 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03ac30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #14 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef97e88,
      func=0x7f0b8a03af50) at ../Python/ceval.c:4471
  #15 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef97e88) at ../Python/ceval.c:4396
  #16 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #17 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a03adb0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #18 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef98068,
      func=0x7f0b8a03f250) at ../Python/ceval.c:4471
  #19 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98068) at ../Python/ceval.c:4396
  #20 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #21 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bd30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a0752a8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  #22 0x0000556bdc325cc2 in function_call (func=0x7f0b8a0771d0, arg=0x7f0b8a3bd5a0, kw=0x7f0b8a042dd0)
      at ../Objects/funcobject.c:523
  #23 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042dd0, arg=0x7f0b8a3bd5a0, func=0x7f0b8a0771d0)
      at ../Objects/abstract.c:2544
  #24 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef982a0, func=0x7f0b8a0771d0)
      at ../Python/ceval.c:4690
  #25 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  #26 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a04bf30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #27 0x0000556bdc325b9b in function_call (func=0x7f0b8a0772d0, arg=arg@entry=0x7f0b8a3bd550, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  #28 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd550, func=<optimized out>) at ../Objects/abstract.c:2544
  #29 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6a00, arg=0x7f0b8a3bd550, kw=0x0)
      at ../Objects/classobject.c:2600
  #30 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6a00, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #31 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cfd0, args=0x7f0b8a044210, kwds=0x0)
      at ../Objects/typeobject.c:5609
  #32 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cfd0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #33 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98798, func=0x7f0b8a03cfd0)
      at ../Python/ceval.c:4593
  #34 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98798) at ../Python/ceval.c:4398
  #35 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #36 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  #37 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd780, kw=0x7f0b8a042cb0)
      at ../Objects/funcobject.c:523
  #38 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042cb0, arg=0x7f0b8a3bd780, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  #39 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef989d0, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  #40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  #41 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #42 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd730, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  #43 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd730, func=<optimized out>) at ../Objects/abstract.c:2544
  #44 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427190, arg=0x7f0b8a3bd730, kw=0x0)
      at ../Objects/classobject.c:2600
  #45 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a427190, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #46 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a03cf90, args=0x7f0b8a044150, kwds=0x0)
      at ../Objects/typeobject.c:5609
  #47 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a03cf90, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #48 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef98ec8, func=0x7f0b8a03cf90)
      at ../Python/ceval.c:4593
  #49 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef98ec8) at ../Python/ceval.c:4398
  #50 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #51 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  #52 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bdaf0, kw=0x7f0b8a042b90)
      at ../Objects/funcobject.c:523
  #53 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042b90, arg=0x7f0b8a3bdaf0, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  #54 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99100, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  #55 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  #56 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #57 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bdb40, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  #58 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bdb40, func=<optimized out>) at ../Objects/abstract.c:2544
  #59 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a3c6820, arg=0x7f0b8a3bdb40, kw=0x0)
      at ../Objects/classobject.c:2600
  #60 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a3c6820, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #61 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c50, args=0x7f0b8a03c0d0, kwds=0x0)
      at ../Objects/typeobject.c:5609
  #62 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c50, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #63 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef995f8, func=0x7f0b8a033c50)
      at ../Python/ceval.c:4593
  #64 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef995f8) at ../Python/ceval.c:4398
  #65 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #66 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d530, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a076ee8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  #67 0x0000556bdc325cc2 in function_call (func=0x7f0b8a07f950, arg=0x7f0b8a3bd500, kw=0x7f0b8a042a70)
      at ../Objects/funcobject.c:523
  #68 0x0000556bdc30fa96 in PyObject_Call (kw=0x7f0b8a042a70, arg=0x7f0b8a3bd500, func=0x7f0b8a07f950)
      at ../Objects/abstract.c:2544
  #69 ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7ffc3ef99830, func=0x7f0b8a07f950)
      at ../Python/ceval.c:4690
  #70 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3052
  #71 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a07d2b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #72 0x0000556bdc325b9b in function_call (func=0x7f0b8a07f850, arg=arg@entry=0x7f0b8a3bd9b0, kw=kw@entry=0x0)
      at ../Objects/funcobject.c:523
  #73 0x0000556bdc33f553 in PyObject_Call (kw=0x0, arg=0x7f0b8a3bd9b0, func=<optimized out>) at ../Objects/abstract.c:2544
  #74 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a4310a0, arg=0x7f0b8a3bd9b0, kw=0x0)
      at ../Objects/classobject.c:2600
  #75 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a4310a0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #76 0x0000556bdc37eff8 in slot_tp_call (self=self@entry=0x7f0b8a033c10, args=0x7f0b8a033fd0, kwds=0x0)
      at ../Objects/typeobject.c:5609
  #77 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a033c10, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #78 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef99d28, func=0x7f0b8a033c10)
      at ../Python/ceval.c:4593
  #79 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99d28) at ../Python/ceval.c:4398
  #80 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #81 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99e38,
      func=0x7f0b8a01ecd0) at ../Python/ceval.c:4461
  #82 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99e38) at ../Python/ceval.c:4396
  #83 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #84 0x0000556bdc3125fe in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef99f48,
      func=0x7f0b8a01edd0) at ../Python/ceval.c:4461
  #85 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef99f48) at ../Python/ceval.c:4396
  #86 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #87 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a081d30, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=1, defs=0x7f0b8a008ba8, defcount=10,
      closure=0x0) at ../Python/ceval.c:3608
  #88 0x0000556bdc325cc2 in function_call (func=0x7f0b8a00ddd0, arg=arg@entry=0x7f0b8a033610, kw=kw@entry=0x7f0b8a014a70)
      at ../Objects/funcobject.c:523
  #89 0x0000556bdc33f553 in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a033610, func=<optimized out>)
      at ../Objects/abstract.c:2544
  #90 instancemethod_call (func=<optimized out>, func@entry=0x7f0b8a427140, arg=0x7f0b8a033610, arg@entry=0x7f0b8a4e0050,
      kw=kw@entry=0x7f0b8a014a70) at ../Objects/classobject.c:2600
  #91 0x0000556bdc33f10f in PyObject_Call (kw=0x7f0b8a014a70, arg=0x7f0b8a4e0050, func=0x7f0b8a427140)
      at ../Objects/abstract.c:2544
  #92 slot_tp_init (self=self@entry=0x7f0b8a3ebb10, args=args@entry=0x7f0b8a4e0050, kwds=kwds@entry=0x7f0b8a014a70)
      at ../Objects/typeobject.c:5869
  #93 0x0000556bdc2feb87 in type_call (type=<optimized out>, type@entry=0x556bdd2a85c0, args=0x7f0b8a4e0050, kwds=0x7f0b8a014a70)
      at ../Objects/typeobject.c:765
  #94 0x0000556bdc2f8903 in PyObject_Call (func=0x556bdd2a85c0, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #95 0x0000556bdc3128dd in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7ffc3ef9a458, func=0x556bdd2a85c0)
      at ../Python/ceval.c:4593
  #96 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a458) at ../Python/ceval.c:4398
  #97 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #98 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d830, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0)
      at ../Python/ceval.c:3608
  #99 0x0000556bdc311886 in PyEval_EvalCode (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, co=<optimized out>)
      at ../Python/ceval.c:669
  #100 exec_statement (locals=0x7f0b8a4ae170, globals=0x7f0b8a4ae170, prog=<optimized out>, f=<optimized out>)
      at ../Python/ceval.c:5093
  #101 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:2122
  #102 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a3ea4b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a381068, defcount=5,
      closure=0x0) at ../Python/ceval.c:3608
  #103 0x0000556bdc312953 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffc3ef9a818,
      func=0x7f0b8a37d8d0) at ../Python/ceval.c:4471
  #104 call_function (oparg=<optimized out>, pp_stack=0x7ffc3ef9a818) at ../Python/ceval.c:4396
  #105 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3013
  #106 0x0000556bdc30b3cc in PyEval_EvalCodeEx (co=0x7f0b8a37d4b0, globals=<optimized out>, locals=<optimized out>,
      args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x7f0b8a379ce8, defcount=1,
      closure=0x0) at ../Python/ceval.c:3608
  #107 0x0000556bdc325b9b in function_call (func=func@entry=0x7f0b8a37de50, arg=0x7f0b8a3b5be0, kw=0x0)
      at ../Objects/funcobject.c:523
  #108 0x0000556bdc2f8903 in PyObject_Call (func=0x7f0b8a37de50, arg=<optimized out>, kw=<optimized out>)
      at ../Objects/abstract.c:2544
  #109 0x0000556bdc3b87e1 in RunModule (module=<optimized out>, set_argv0=1) at ../Modules/main.c:197
  #110 0x0000556bdc3a68ed in Py_Main (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:592#111 0x00007f0b8a54bd0a in __libc_start_main (main=0x556bdc3a6530 <main>, argc=5, argv=0x7ffc3ef9ac08, init=<optimized out>,
      fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc3ef9abf8) at ../csu/libc-start.c:308
  #112 0x0000556bdc3a646a in _start ()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants