Can greenlet work with the GLib event loop? It segfaults :(

[wusspuss]

I'm looking to resume a running greenlet from inside a GLib event loop callback (any such). To demonstrate we can use GLib.idle_add

import gi
gi.require_version("GLib", "2.0")
from gi.repository import GLib
from greenlet import greenlet

main_greenlet = greenlet.getcurrent()
def my_thread():
    for i in range(10):
        print(i)
        main_greenlet.switch()
mt = greenlet(my_thread)

loop = GLib.MainLoop()
GLib.idle_add(mt.switch)
loop.run()

I get '0' printed and then a segfault. Is there any possible workaround?

#0  PyObject_IsTrue (v=v@entry=0x0) at Objects/object.c:1506
#1  0x00007ffff6fe33f6 in pygi_gboolean_from_py (result=0x7fffffffd688, object=0x0) at ../pygobject/gi/pygi-basictype.c:436
#2  pygi_marshal_from_py_basic_type
    (object=0x0, arg=0x7fffffffd688, type_tag=<optimized out>, transfer=<optimized out>, cleanup_data=0x5555557086c0)
    at ../pygobject/gi/pygi-basictype.c:1051
#3  0x00007ffff6fd67e9 in _pygi_closure_set_out_arguments
    (resp=0x7fffffffd890, py_retval=0x7ffff7d73ff8 <_PyRuntime+58904>, cache=0x555555705a90, state=0x7fffffffd660)
    at ../pygobject/gi/pygi-closure.c:442
#4  _pygi_closure_handle
    (cif=<optimized out>, result=result@entry=0x7fffffffd890, args=args@entry=0x7fffffffd700, data=data@entry=0x5555556d9140)
    at ../pygobject/gi/pygi-closure.c:590
#5  0x00007ffff7de2152 in ffi_closure_unix64_inner
    (cif=<optimized out>, fun=<optimized out>, user_data=<optimized out>, rvalue=<optimized out>, reg_args=<optimized out>, argp=0x7fffffffd8c0 "\020\275\367\366\377\177") at ../src/x86/ffi64.c:899
#6  0x00007ffff7de27b8 in ffi_closure_unix64 () at ../src/x86/unix64.S:303
#7  0x00007ffff6eb9f69 in g_main_dispatch (context=0x5555556ff780) at ../glib/glib/gmain.c:3476
#8  0x00007ffff6f183a7 in g_main_context_dispatch_unlocked (context=0x5555556ff780) at ../glib/glib/gmain.c:4284
#9  g_main_context_iterate_unlocked.isra.0
    (context=0x5555556ff780, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/glib/gmain.c:4349
#10 0x00007ffff6ebab97 in g_main_loop_run (loop=0x555555704f80) at ../glib/glib/gmain.c:4551
#11 0x00007ffff7de2596 in ffi_call_unix64 () at ../src/x86/unix64.S:104
#12 0x00007ffff7ddf00e in ffi_call_int
--Type <RET> for more, q to quit, c to continue without paging--
 at ../src/x86/ffi64.c:673
#13 0x00007ffff7de1bd3 in ffi_call (cif=cif@entry=0x5555557082f8, fn=<optimized out>, rvalue=rvalue@entry=0x7fffffffdc08, avalue=<optimized out>) at ../src/x86/ffi64.c:710
#14 0x00007ffff6fd76d1 in pygi_invoke_c_callable (function_cache=0x555555708250, state=<optimized out>, py_args=<optimized out>, py_kwargs=<optimized out>) at ../pygobject/gi/pygi-invoke.c:684
#15 0x00007ffff6fd6090 in pygi_function_cache_invoke (py_kwargs=0x0, py_args=0x7ffff712a4d0, function_cache=<optimized out>) at ../pygobject/gi/pygi-cache.c:862
#16 pygi_callable_info_invoke (user_data=0x0, cache=<optimized out>, kwargs=0x0, py_args=0x7ffff712a4d0, info=<optimized out>) at ../pygobject/gi/pygi-invoke.c:727
#17 _wrap_g_callable_info_invoke (self=<optimized out>, py_args=0x7ffff712a4d0, kwargs=0x0) at ../pygobject/gi/pygi-invoke.c:764
#18 0x00007ffff6fc90aa in _callable_info_call (kwargs=0x0, args=0x7ffff7d73ff8 <_PyRuntime+58904>, self=0x7ffff6361930) at ../pygobject/gi/pygi-info.c:548
#19 _callable_info_call (self=0x7ffff6361930, args=0x7ffff7d73ff8 <_PyRuntime+58904>, kwargs=0x0) at ../pygobject/gi/pygi-info.c:525
#20 0x00007ffff79dc054 in _PyObject_MakeTpCall (tstate=0x7ffff7d8e398 <_PyRuntime+166328>, callable=0x7ffff6361930, args=<optimized out>, nargs=0, keywords=0x0) at Objects/call.c:214
#21 0x00007ffff79e76e1 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:4769
#22 0x00007ffff7aa0954 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7f9d020, tstate=0x7ffff7d8e398 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73
#23 _PyEval_Vector (tstate=tstate@entry=0x7ffff7d8e398 <_PyRuntime+166328>, func=func@entry=0x7ffff75f2020, locals=locals@entry=0x7ffff7612d00, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0)
    at Python/ceval.c:6434
#24 0x00007ffff7aa033c in PyEval_EvalCode (co=0x7ffff7561110, globals=<optimized out>, locals=0x7ffff7612d00) at Python/ceval.c:1148
#25 0x00007ffff7abde43 in run_eval_code_obj (tstate=tstate@entry=0x7ffff7d8e398 <_PyRuntime+166328>, co=co@entry=0x7ffff7561110, globals=globals@entry=0x7ffff7612d00, locals=locals@entry=0x7ffff7612d00) at Python/pythonrun.c:1710
#26 0x00007ffff7ab9f7a in run_mod
    (mod=mod@entry=0x5555555f3a98, filename=filename@entry=0x7ffff75c0530, globals=globals@entry=0x7ffff7612d00, locals=locals@entry=0x7ffff7612d00, flags=flags@entry=0x7fffffffe188, arena=arena@entry=0x7ffff753b7b0)
    at Python/pythonrun.c:1731
#27 0x00007ffff7ad04b3 in pyrun_file
    (fp=fp@entry=0x555555559370, filename=filename@entry=0x7ffff75c0530, start=start@entry=257, globals=globals@entry=0x7ffff7612d00, locals=locals@entry=0x7ffff7612d00, closeit=closeit@entry=1, flags=0x7fffffffe188)
    at Python/pythonrun.c:1626
#28 0x00007ffff7acfe25 in _PyRun_SimpleFileObject (fp=0x555555559370, filename=0x7ffff75c0530, closeit=1, flags=0x7fffffffe188) at Python/pythonrun.c:440
#29 0x00007ffff7ace728 in _PyRun_AnyFileObject (fp=0x555555559370, filename=0x7ffff75c0530, closeit=1, flags=0x7fffffffe188) at Python/pythonrun.c:79
#30 0x00007ffff7ac91d8 in pymain_run_file_obj (skip_source_first_line=0, filename=0x7ffff75c0530, program_name=0x7ffff7617b40) at Modules/main.c:360
#31 pymain_run_file (config=0x7ffff7d743e0 <_PyRuntime+59904>) at Modules/main.c:379
#32 pymain_run_python (exitcode=0x7fffffffe180) at Modules/main.c:601
#33 Py_RunMain () at Modules/main.c:680
#34 0x00007ffff7a940db in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734
#35 0x00007ffff7643cd0 in __libc_start_call_main (main=main@entry=0x555555555120 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffe3d8) at ../sysdeps/nptl/libc_start_call_main.h:58
#36 0x00007ffff7643d8a in __libc_start_main_impl (main=0x555555555120 <main>, argc=2, argv=0x7fffffffe3d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe3c8) at ../csu/libc-start.c:360
#37 0x0000555555555045 in _start ()

[jamadden]

Conceptually, there's no reason why it shouldn't work (after all, that's essentially what gevent does) --- provided you adhere to the rules of the event loop. It looks like this event loop is very much not reentrant. That is, if we linearize the code-flow of your example, it would be something like this:

running event loop <- implemented in C
    running idle callbacks  <- implemented in C
        for cb in callbacks: <- implemented in C
            return_value = cb() <- implemented in Python
                         ^ Where it goes wrong 
                switch to another greenlet 
                    switch right back to here
                         return_value = ?????

In other words, you're switching back into the main greenlet while it is still running your callback. The main greenlet expected a simple function call that returns a value, but that didn't happen. So when the main greenlet starts running again, it's in an indeterminate state --- there's no return value to be had.

Cases like this can often be handled by using more callbacks, although it's not very pretty. gevent had to do something like that to work with libuv, which is also not reentrant; we wound up with a single callback that just queues other callbacks that we execute after one iteration of the event loop.

while True:
     run event loop once 
        for cb in callbacks:
               gevent callback
                     collect ready greenlets; store them
    run stored callbacks

I'm completely unfamiliar with the internals of the library you're working with, so this is all informed conjecture based on the function names in the stack trace.