Deadlock with python 3.11

[epizut]

I spent a lot of time digging into a deadlock issue with Dask distributed and Python 3.11.
This issue was created so other people won't have to lose the same amount of time debugging endless dask and CPython stacks.

Thanks to @diegorusso we now have a pending fix python/cpython#116969

Solutions:

• Wait for #116969 if you are forced to use py3.11, but be aware that you need either python<3.11.9 or dask>=2024.4.1 due to #11035
• Skip 3.11 and move to 3.12

[fjetter]

Thanks a lot for tracking this down and reporting it here!

[diegorusso]

It really depends if they are going to accept the PR. In theory 3.11.9 is the latest release bug fix they do it. After that, there will be just source distributions releases with security fixes for the distros to pick up.
I really hope they accept the PR because 3.11 will still be used for quite some time.

[epizut]

Sure, thanks again for your work on this.
In the meamtime, we could make the dask gc diagnostic optional or even better try to make it non re-entrant on the CPython lock by avoiding calling _current_frames()