Add support for Python 3.11

[graingert]

• Closes Python 3.11 support  #9691
• Tests added / passed
• Passes pre-commit run --all-files

[graingert]

getting a segfault here: https://github.com/dask/dask/actions/runs/3638328422/jobs/6140402545#step:7:11195 looks like pandas-dev/pandas#50105

[jrbourbeau]

Nice detective work @graingert. Sounds like we can safely xfail the unhappy hdf-related tests on Python 3.11

[graingert]

we can't xfail them as they segfault - will have to skip them for now

[jrbourbeau]

👍

[graingert]

depends on dask/distributed#7249

[jrbourbeau]

Restarting failed builds as it looks like the linux Python 3.11 build stalled for some reason

[jakirkham]

Should this still be a draft?

[graingert]

Hmm pyarrow isn't working due to

Encountered problems while solving:

• package pyarrow-10.0.1-py310he8a299c_0_cuda requires libarrow 10.0.1 ha344585_0_cuda, but none of the providers can be installed

[jakirkham]

Would just pin to Arrow 9. Likely it will require work to upgrade to a newer version

[crusaderky]

Isn't it easier to comment it out again for the time being? numba is also unavailable anyway and they have a history of being several months late every time there's a new major python version.

[jakirkham]

There should be a Python 3.11 package for Arrow 9 already

It is just Arrow 10 came out somewhat recently and there are some fairly significant changes in it that should improve things overall. Just will take time to fully adapt to. So that's what we are seeing here

[crusaderky]

@graingert commented in a previous version of the PR that pyarrow 10 is necessary for python 3.11

[jakirkham]

That doesn't sound right. Arrow 9 was definitely built for Python 3.11 (for example). If that doesn't work, would be very interested to know.

[jrbourbeau]

Okay, looks like the issue was due to the libprotobuf pin. That's only needed for the pyarrow-nightly build and has since been moved to a different file anyways. After removing the libprotobuf pin, I was able to get a solved Python 3.11 environment locally on my mac. Let's see if it solves in CI.

[jrbourbeau]

Looks like that did the trick

[jrbourbeau]

The Linux 3.11 build stalled for some reason, so I just retriggered CI to see if the stall happens consistently or not. If it doesn't, I think this is good to go

[mrocklin]

I'll be glad to see this get in. Thank you everyone who helped here. I'm excited for Python 3.11.

[jrbourbeau]

Okay, looks like the hanging Linux Python 3.11 build passed (🎉 ) but we're seeing dask/tests/test_multiprocessing.py::test_custom_context_used_python3_posix fail on macOS Python 3.11. We're also seeing it fail consistently. I'm not able to reproduce locally when I run that test in isolation. Looking into it now... Would certainly welcome thoughts from others too : )

[jrbourbeau]

Alright, looks like tiledb may have been the source of the issue. At least locally when I run the tiledb-related tests alongside test_custom_context_used_python3_posix with pytest dask -k "test_custom_context_used_python3_posix or tiledb" I'm able to reproduce the error. I've temporarily commented tiledb out of the Python 3.11 CI environment file. 🤞 that fixes things

[jrbourbeau]

Thanks @graingert @crusaderky @jakirkham!