Python 3.11 support #11170

kenodegard · 2022-01-27T19:52:29Z

Description

We need to investigate the changes necessary to support Python 3.11. Because this release cycle happens every year. It would also make sense for us to have a policy in place guiding the work that we need to regularly complete for this.

Good to know

There is currently a rough draft of this document that could be used as a starting point for the work:
conda/ceps#24

Tasks

Give feedback

Python 3.11 #12256

cla-signed locked source::community
Create Python 3.11 image #12580

cla-signed locked
Options

EwoutH · 2022-05-17T10:23:04Z

Python 3.11 is now in beta, which means no new features will be added.

travishathaway · 2022-05-31T11:01:24Z

In response to this issue, I have created the following draft CEP (Conda Enhancement Proposal):

conda/ceps#24

After finishing and adopting this CEP, conda will have an official policy surrounding new (C)Python version adoption.

EwoutH · 2022-10-10T11:23:27Z

Any update on this effort? Python 3.11 is now in the release candidate phase and will be released in two weeks. Many projects already support Python 3.11 and have wheels available, including NumPy, Pandas, SciPy, Matplotlib and Seaborn.

travishathaway · 2022-10-17T13:21:06Z

@EwoutH, thanks for checking in. This issue is high on our priority list, and we hope to begin working on it in an upcoming sprint. I can not say for certain when this will be completed though.

EwoutH · 2022-10-18T09:27:40Z

Thanks for getting back! Great to hear the team is working on this, keep us in the loop :)

johnnynunez · 2022-10-25T13:08:13Z

Yeah! Python 3.11 support please. It should be the default.
Python 3.11 is between 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite. See Faster CPython for details.
https://docs.python.org/3.11/whatsnew/3.11.html
https://www.phoronix.com/news/Python-3.11-Released

soumendrak · 2022-10-25T19:13:41Z

Python 3.11.0 support please 🙏🏼

johnnynunez · 2022-10-26T11:10:31Z

Benchmark https://www.phoronix.com/review/python-311-performance

EwoutH · 2022-10-27T22:46:03Z

Hi @travishathaway! Do you have anything to share as of this moment?

kingmo888 · 2022-10-31T06:12:15Z

Anxious expectation python 3.11.0

EwoutH · 2022-12-15T06:25:35Z

We're now over 4 months over rc1 and almost two months after stable release. It's blocking many major projects. Could we get an update on 3.11 support?

Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues. ### Main changes - Upgrade cython to 0.29.32 - Add CI/CD steps for 3.11 wheel - Change cython code to not use `recursion_depth` - Update cloudpickle to latest - Use newer manylinux2014 which has python3.11 - Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version ### Checklist: - cython - [x] remove deprecated `recursion_depth` - [x] exc_type cython/cython#4500 - [ ] package dependencies https://pyreadiness.org/3.11/ - [ ] llvmlite numba/llvmlite#869 - [ ] numba numba/numba#8304 - [ ] pyarrow - [ ] scikit-learn - [ ] pydantic - [x] cloudpickle - [x] upgrade cython to 0.29.32 - [ ] tensorflow tensorflow/tensorflow#58032 - [ ] torch pytorch/pytorch#86566 - [ ] miniconda conda/conda#11170 - [ ] `docker/base-deps/Dockerfile` - [x] claim to support 3.11 in setup.py - [ ] cicd - [ ] .buildkite/ - [ ] .buildkite/pipeline.build.yml - [ ] ci/ - [ ] ci/build/test-wheels.sh - [ ] ci/build/build-docker-images.py - [ ] release tests - [ ] docker/retag-lambda/python_versions.txt - [ ] download_wheels.sh - [ ] wheels - [ ] `python/build-wheel-macos.sh` - [ ] `python/build-wheel-windows.sh` - [ ] Tests - [ ] pytest ray/serve/tests - [ ] python python/ray/serve/examples/echo_full.py - [ ] bazel test //:core_worker_test - [ ] bazel test --build_tests_only //:all - [ ] //python/ray/tests:test_pydantic_serialization tiangolo/fastapi#5048 - [ ] //python/ray/train:test_torch_utils - [ ] Documentation - [x] installation.rst Current status: Linux and mac wheels build in CICD. Docker images will come in a separate PR.

RNarayan73 · 2023-03-13T13:20:41Z

Due to be released on March 14, fingers crossed!
https://www.anaconda.com/blog/upcoming-releases-anaconda-distribution-2023-03-and-beyond#:~:text=Python%203.11%20support%20will%20be,of%20Anaconda's%20biggest%20releases%20yet!

rickyyx · 2023-03-16T18:41:03Z

Didn't see relevant info of 3.11 in the release note: https://github.com/conda/conda/releases/tag/23.3.0

But are we having 3.11 support officially?

johnnynunez · 2023-03-16T19:12:08Z

Didn't see relevant info of 3.11 in the release note: https://github.com/conda/conda/releases/tag/23.3.0

But are we having 3.11 support officially?

there are binaries for python 3.11 but not all

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com> Signed-off-by: Jack He <jackhe2345@gmail.com>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com>

johnnynunez · 2023-03-22T16:02:16Z

When will the python3.11 image be available for testing? ghcr.io/conda/conda-ci:main-linux-python3.11

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com> Signed-off-by: chaowang <chaowang@anyscale.com>

kenodegard · 2023-04-17T22:25:28Z

Python 3.11 support added via #12256, this will be included in the May release of conda 23.5.0

EwoutH · 2023-04-18T05:24:25Z

Awesome, great to hear!

Maybe this process can now be documented, while it's still fresh in everyones memory, so that it doesn't take so long for future Python releases.

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com> Signed-off-by: elliottower <elliot@elliottower.com>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment) It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds. A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24 A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d Signed-off-by: rickyyx <rickyx@anyscale.com> Co-authored-by: scv119 <scv119@gmail.com> Signed-off-by: Jack He <jackhe2345@gmail.com>

Since miniconda hasn't released Python 3.11 support yet as per conda/conda#11170 (comment)

* Add initial test workflow * Add decorator as test dependency * Move test dependencies into extras_require * Add environment.yml with packages taken from .travis.yml * Limit concurrency * Only trigger on PRs to default branch * Only upgrade packages if needed * Use conda-incubator/setup-miniconda instead of actions/setup-python * Remove atlas * Rename test.yaml to test.yml * `python -m pip` * Use login shell so conda environment gets activated * Remove dependencies * Remove upper bound on matplotlib * Expand build matrix 1. Use both flexible and strict channel priority for conda environment. 2. Include macOS and Windows for latest Python version. * Set use-only-tar-bz2 for caching As per https://github.com/conda-incubator/setup-miniconda/tree/v2/#caching-packages * Remove redundant dependency mention SciPy is already part of install_requires so not necessary to have it in extras_require as well. * mpl_ic.image_comparison -> pytest.mark.mpl_image_compare * Use Python 3.10 as the latest version Since miniconda hasn't released Python 3.11 support yet as per conda/conda#11170 (comment) * Install pytest with conda * Log package and dependency information

kenodegard added source::anaconda created by members of Anaconda, Inc. type::task indicates a change that doesn't pertain to the code itself, e.g. updating CI/CQ, rebuilding package ¡important! used to indicate particularly notable information labels Jan 27, 2022

jezdez added this to the Release TBD milestone Feb 8, 2022

kenodegard added sync::anaconda sync internally with Anaconda, Inc. ticket tracker backlog issue has been triaged but has not been earmarked for any upcoming release labels Feb 10, 2022

jezdez mentioned this issue Mar 29, 2022

Add python constraint back for --update-deps --only-deps #11378

Merged

EwoutH mentioned this issue May 17, 2022

CI: Add Python 3.11 jobs to matrix #11481

Closed

awwad changed the title ~~Python 3.11 support~~ CEP Drafting for Python 3.11 support Jul 20, 2022

kenodegard mentioned this issue Aug 22, 2022

Python 3.11 support conda/conda-build#4370

Closed

travishathaway self-assigned this Sep 21, 2022

kenodegard changed the title ~~CEP Drafting for Python 3.11 support~~ Python 3.11 support Sep 29, 2022

This was referenced Oct 10, 2022

Release 22.11.X #11932

Closed

ENH: Support Python 3.11 geopandas/geopandas#2592

Closed

johnnynunez mentioned this issue Dec 21, 2022

Add support for Python 3.11 pytorch/vision#7049

Open

csko mentioned this issue Dec 25, 2022

Prepare Python 3.11 support ray-project/ray#31170

Merged

43 tasks

malfet mentioned this issue Jan 4, 2023

Python 3.11.1 , even with nightly version of PyTorch: ERROR: No matching distribution found for torch pytorch/pytorch#91543

Closed

kenodegard added the duplicate::primary if an issue/PR has duplicates, this is the consolidated, primary issue/PR label Mar 6, 2023

kenodegard modified the milestones: 23.3.0, 23.5.0 Mar 14, 2023

rickyyx mentioned this issue Mar 16, 2023

[CI][runtime-env] fix runtime-env test ray-project/ray#33400

Merged

7 tasks

travishathaway assigned jezdez and unassigned travishathaway Mar 20, 2023

kenodegard added in-progress issue is actively being worked on and removed backlog issue has been triaged but has not been earmarked for any upcoming release labels Mar 22, 2023

kenodegard mentioned this issue Apr 11, 2023

Python 3.11 #12256

Merged

kenodegard closed this as completed Apr 17, 2023

rickyyx mentioned this issue Apr 23, 2023

[ci][python3.11] Wheel build ray-project/ray#31362

Closed

carlthome added a commit to carlthome/mir_eval that referenced this issue Jun 15, 2023

Use Python 3.10 as the latest version

10dbf30

Since miniconda hasn't released Python 3.11 support yet as per conda/conda#11170 (comment)

github-actions bot added the locked [bot] locked due to inactivity label Apr 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.11 support #11170

Python 3.11 support #11170

kenodegard commented Jan 27, 2022 •

edited

Tasks

EwoutH commented May 17, 2022

travishathaway commented May 31, 2022

EwoutH commented Oct 10, 2022

travishathaway commented Oct 17, 2022

EwoutH commented Oct 18, 2022

johnnynunez commented Oct 25, 2022

soumendrak commented Oct 25, 2022

johnnynunez commented Oct 26, 2022

EwoutH commented Oct 27, 2022

kingmo888 commented Oct 31, 2022

EwoutH commented Dec 15, 2022 •

edited

RNarayan73 commented Mar 13, 2023

rickyyx commented Mar 16, 2023

johnnynunez commented Mar 16, 2023

johnnynunez commented Mar 22, 2023 •

edited

kenodegard commented Apr 17, 2023

EwoutH commented Apr 18, 2023

Python 3.11 support #11170

Python 3.11 support #11170

Comments

kenodegard commented Jan 27, 2022 • edited

Description

Good to know

Tasks

EwoutH commented May 17, 2022

travishathaway commented May 31, 2022

EwoutH commented Oct 10, 2022

travishathaway commented Oct 17, 2022

EwoutH commented Oct 18, 2022

johnnynunez commented Oct 25, 2022

soumendrak commented Oct 25, 2022

johnnynunez commented Oct 26, 2022

EwoutH commented Oct 27, 2022

kingmo888 commented Oct 31, 2022

EwoutH commented Dec 15, 2022 • edited

RNarayan73 commented Mar 13, 2023

rickyyx commented Mar 16, 2023

johnnynunez commented Mar 16, 2023

johnnynunez commented Mar 22, 2023 • edited

kenodegard commented Apr 17, 2023

EwoutH commented Apr 18, 2023

kenodegard commented Jan 27, 2022 •

edited

EwoutH commented Dec 15, 2022 •

edited

johnnynunez commented Mar 22, 2023 •

edited