Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix several greenlet leaks #261

Merged
merged 65 commits into from
Oct 13, 2021
Merged

Fix several greenlet leaks #261

merged 65 commits into from
Oct 13, 2021

Commits on Sep 29, 2021

  1. A bare Event.wait() with no timeout is an invitation to hang the proc…

    …ess.
    
    Add a timeout to all such instances.
    jamadden committed Sep 29, 2021
    Configuration menu
    Copy the full SHA
    54105f0 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2021

  1. See if all compilers can work in C++ mode.

    Thread-local C++ objects should be a portable way to get thread destructors.
    
    The slp_* functions need to be compiled as C because we call them from assembly.
    
    forward declarations should also be c linkage, otherwise MSVC complains.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    24e4cba View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e960906 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1921580 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8dcf174 View commit details
    Browse the repository at this point in the history
  5. Port all global variables to a threadlocal struct.

    This eliminates the complexity of green_updatecurrent, I believe. Much more can be simplified as well.
    
    Still need to add the thread cleanup functions.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    82be175 View commit details
    Browse the repository at this point in the history
  6. Add benchmarks.

    Currently, we're somewhat slower than the global variable case, as
    expected.
    
    | Benchmark                    | gl-1.1.2 | tl1                   |
    |------------------------------|:--------:|:---------------------:|
    | create a greenlet            | 68.7 ns  | 88.0 ns: 1.28x slower |
    | switch between two greenlets | 301 ns   | 435 ns: 1.45x slower  |
    | getcurrent single thread     | 17.9 ns  | 31.8 ns: 1.78x slower |
    | chain(100000)                | 256 ms   | 275 ms: 1.08x slower  |
    | Geometric mean               | (ref)    | 1.37x slower          |
    
    Also some Windows debugging.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    ff3c7c2 View commit details
    Browse the repository at this point in the history
  7. debug.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    024d3d7 View commit details
    Browse the repository at this point in the history
  8. more debugging.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    1b7906c View commit details
    Browse the repository at this point in the history
  9. g_switchstack was using a stack local variable, which it can't do.

    Amusingly, this only caused issues on 64-bit Python 3.9 and 3.10 an Windows.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    7e24943 View commit details
    Browse the repository at this point in the history
  10. Perhaps MSVC was combining the two apparent copies of g_thread_state_…

    …global in g_switchstack into a single copy, thus messing up the stack.
    
    Try accessing it with the global variable each time.
    
    This makes clang, at least, look it up each time, even with optimizations on.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    be25e20 View commit details
    Browse the repository at this point in the history
  11. Try a separate function.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    2cd7d74 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    c79a40a View commit details
    Browse the repository at this point in the history
  13. Checkpoint on #252.

    We're now using Py_AddPendingCall to clean up some thread-local things that don't clean themselves up. I'm still working to be sure we break cycles.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    d59bb5b View commit details
    Browse the repository at this point in the history
  14. Fix compilation on MSVC.

    You need special syntax to return a pointer in a noinline function in MSVC.
    
    Because the parser ignores, without warning, __declspec after a * character.
    https://docs.microsoft.com/en-us/cpp/cpp/declspec?view=msvc-160
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    2f22881 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    caf72df View commit details
    Browse the repository at this point in the history
  16. You can't call PyObject_Print if there's an error set.

    Improve the debugging to not do that. Also limit it to a suspected issue for now.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    32e7124 View commit details
    Browse the repository at this point in the history
  17. More debugging

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    cb14308 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    0b19e32 View commit details
    Browse the repository at this point in the history
  19. Checkpoint moving to a more flexible use of TLS.

    This should let us restore support for older compilers.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    9ad0d55 View commit details
    Browse the repository at this point in the history
  20. The indirect TLS support appears to be working, and I have fixed #252

    I haven't tested this on other platforms or with a more complex app like gevent yet though.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    340664f View commit details
    Browse the repository at this point in the history
  21. Some debugging.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    5e5fa2a View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    3a2a33d View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    da7f417 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    cf4f729 View commit details
    Browse the repository at this point in the history
  25. Queue pending cleanups to reduce the number of Py_AddPendingCall slot…

    …s we take.
    
    Also add a testing API to be sure cleanups have finished.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    a421362 View commit details
    Browse the repository at this point in the history
  26. Separating out concerns.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    f80d0fb View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    3211747 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    a22f68e View commit details
    Browse the repository at this point in the history
  29. Use the old idiom.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    62fac36 View commit details
    Browse the repository at this point in the history
  30. Stop including headers we know aren't present. We got to that point o…

    …f the compile on old MSVC.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    3662fba View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    9256f4c View commit details
    Browse the repository at this point in the history
  32. MFC was a no-go. So use the win32 API.

    may need to force linking with kernel32
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    6c350ef View commit details
    Browse the repository at this point in the history
  33. Introduce PyMainGreenlet as a subtype of greenlet.

    This way we don't have to grow the greenlet class by a pointer that is almost always empty. Also brings more type safety to the code.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    4a35c59 View commit details
    Browse the repository at this point in the history
  34. Fix bad descriptor error on Python 2.7 for the main greenlet, and com…

    …pilation error on MSVC.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    e25aea0 View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    5bf7906 View commit details
    Browse the repository at this point in the history
  36. Add GHA test cases that disable the use of standard threading.

    It passes for me locally.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    9b94af6 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    fadae21 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    bcaf0f1 View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    a7dbfe4 View commit details
    Browse the repository at this point in the history
  40. Passing the thread state through all the calls was too much, see if w…

    …e can save just this one.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    c7841e3 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    04c9166 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    6061284 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    99d7f0d View commit details
    Browse the repository at this point in the history
  44. Fix a faulty assertion.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    af9b38c View commit details
    Browse the repository at this point in the history
  45. Tweak compiler flags.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    0792917 View commit details
    Browse the repository at this point in the history
  46. More Py2/Py3 cleanups.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    5aee563 View commit details
    Browse the repository at this point in the history
  47. Py2.7 testing.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    7f31618 View commit details
    Browse the repository at this point in the history
  48. Remove more debugging code.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    4ec6c35 View commit details
    Browse the repository at this point in the history
  49. Update documentation.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    d18ac12 View commit details
    Browse the repository at this point in the history
  50. Ensure more tests don't leak greenlets.

    Unfortunately, one still does and we have to manually cleanup a few others. Those can be fixed later.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    0da2574 View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    b8a8a84 View commit details
    Browse the repository at this point in the history
  52. Debugging

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    6358768 View commit details
    Browse the repository at this point in the history
  53. More debugging

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    1a28424 View commit details
    Browse the repository at this point in the history
  54. Yet more debugging.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    4099794 View commit details
    Browse the repository at this point in the history
  55. Yet more debugging.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    3d155fa View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    33c667c View commit details
    Browse the repository at this point in the history
  57. Configuration menu
    Copy the full SHA
    831ed82 View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    bef2234 View commit details
    Browse the repository at this point in the history
  59. Configuration menu
    Copy the full SHA
    291c7a1 View commit details
    Browse the repository at this point in the history
  60. Always use RAII/stack-based locking/unlocking.

    This is less error prone.
    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    893844a View commit details
    Browse the repository at this point in the history
  61. Configuration menu
    Copy the full SHA
    e342bda View commit details
    Browse the repository at this point in the history
  62. Configuration menu
    Copy the full SHA
    eb61b4c View commit details
    Browse the repository at this point in the history
  63. Configuration menu
    Copy the full SHA
    cf3637d View commit details
    Browse the repository at this point in the history
  64. Add change note.

    jamadden committed Oct 12, 2021
    Configuration menu
    Copy the full SHA
    7d85029 View commit details
    Browse the repository at this point in the history