Killing greenlets across threads may leak greenlets and their stacks?

[jamadden]

If proper APIs aren't called in the target thread. See #245 (comment)

The analysis there may not actually be correct. There is a reference cycle, but the greenlets contained in the deleteme list should be subject to garbage collection (eventually) because none of them should be active after the thread dies.

[jamadden]

Similarly, there's no cleanup of the main greenlet from a thread unless some other thread calls one of those proper APIs. The main greenlet from a thread is left "dangling" in a suspended state until the user explicitly updates the state from a different thread. For example, this always reports a growth in greenlets:

import gc
import threading

import greenlet

def count_objects(kind=list):
    # pylint:disable=unidiomatic-typecheck
    # Collect the garbage.
    for _ in range(3):
        gc.collect()
    gc.collect()
    return sum(
        1
        for x in gc.get_objects()
        if type(x) is kind
    )

def target():
    print(greenlet.getcurrent())

print("Begin glets", count_objects(greenlet.greenlet))
t = threading.Thread(target=target)
t.start()
t.join()
del t
print("End glets", count_objects(greenlet.greenlet))

$ python foo.py
Begin glets 1
End glets 2

Inserting a call to greenlet.getcurrent() after del t resolves the issue.

[jamadden]

With #251 fixed (when you call getcurrent() in both threads), the list still leaks if you don't call getcurrent() in the background thread before it exits. Both the main and secondary greenlet from the background thread are still reachable by GC, GC is in fact traversing into them both, and yet no cycles are broken (green_clear is never called).

[jamadden]

Sigh. I have a branch that partly fixes this (I can solve the list leak), but there are still cases where it's very difficult to fix completely. If you start a greenlet in a background thread, provide it to another thread, don't let the background greenlet complete, and don't have the background thread call a greenlet API before finishing and after the last reference to the background greenlet has been deleted, then at least the main greenlet from the background thread leaks (or more, depending on the parent chain).

See

greenlet/src/greenlet/tests/test_leaks.py

Line 89 in 5d76ab4

def test_issue251_killing_cross_thread_leaks_list(self, manually_collect_background=True):

, as called (passing False for the parameter) by

greenlet/src/greenlet/tests/test_leaks.py

Line 170 in 5d76ab4

def test_issue251_issue252_need_to_collect_in_background(self):

Currently, the frames from the secondary background greenlet also leak. My branch can at least fix that much. It may not be possible to fix the whole thing though, because a leaking reference is in the C stack and inaccessible. I still have a few things to try, though.