Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare Python 3.11 support #31170

Merged
merged 52 commits into from
Jan 7, 2023
Merged

Prepare Python 3.11 support #31170

merged 52 commits into from
Jan 7, 2023

Conversation

csko
Copy link
Contributor

@csko csko commented Dec 17, 2022 •
edited

Why are these changes needed?

Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues.

Main changes

  • Upgrade cython to 0.29.32
  • Add CI/CD steps for 3.11 wheel
  • Change cython code to not use recursion_depth
  • Update cloudpickle to latest
  • Use newer manylinux2014 which has python3.11
  • Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version

Checklist:

Current status:
Linux and mac wheels build in CICD. Docker images will come in a separate PR.

Related issue number

#27881

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@csko csko mentioned this pull request Dec 17, 2022
Closed
@csko csko force-pushed the py311 branch 2 times, most recently from 8f42e7e to 9cb75eb Compare December 17, 2022 06:04
@csko
python3.11 prep
7a18111 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
fix recursion limit
9255b7a 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
update cicd, buildkite
65e77a9 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
upgrade cython to 0.29.32
d5fa276 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
Copy link
Contributor Author

csko commented Dec 18, 2022 •
edited

Some success building locally:

(venv) csko@cskos-MacBook-Air python % pip install -e . --verbose
...
Successfully installed ray-3.0.0.dev0
(venv) csko@cskos-MacBook-Air python % python
Python 3.11.0 (main, Oct 26 2022, 19:06:18) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>>

@csko
pydantic
4adad69 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
update cloudpickle
814581b 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
condition recursion_depth on <3.11
5ccaa98 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
lint
1f0015c 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
condition pip packages on py311
01dcb87 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
fix wheels
1d9daf8 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@rickyyx rickyyx self-assigned this Dec 21, 2022
@csko
lint
466a835 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
add 3.11 to wheel scripts
ddc700c 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
add numpy version to wheel script
f751360 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
more specific pydantic versions
4d33b4f 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
use newer pypa images with 311
9726e0a 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
fix macos build
f33ea25 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
Merge branch 'ray-project:master' into py311
350a69f
@csko
add 311 to wheel test
7dc33cd 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
try build without 311
fb05c4a 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
Revert "use newer pypa images with 311"
90cb548 
This reverts commit 9726e0a.

Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@rickyyx rickyyx assigned simon-mo and unassigned scv119 Jan 6, 2023
@csko
remove other commented-out 3.11 references
00d7138 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
remove gitpod (merge upstream)
f22eaab 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
@csko
Merge branch 'ray-project:master' into py311
513b97b
@csko
Copy link
Contributor Author

csko commented Jan 6, 2023

  • updated PR with a description of the major changes
  • merged with upstream
  • removed commented out code

@rickyyx
Copy link
Contributor

rickyyx commented Jan 6, 2023 •
edited

Thanks so much. LGTM!

A few notes on test for other reviewers:

  1. Lint check is soft failing since the wheel from master branch hasn't been uploaded, so the URL in the doc is not valid yet. This should be auto-resovled once the PR is merged and CI runs on the master.
  2. Previous mac run see here: https://buildkite.com/ray-project/oss-ci-build-pr/builds/8659#01858375-6bb5-4733-b638-4a1d2c5a25ea
  3. I have manually tested the built wheels on Linux with Ray quick start installation, and we are tracking some incompatibility here: https://github.com/ray-project/ray/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Python+3.11+%22

rickyyx
rickyyx approved these changes Jan 6, 2023
View reviewed changes
Copy link
Contributor

@rickyyx rickyyx left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also - @csko could you update the PR discription (title) to reflect what we have left out in the PR and what have been included? Thanks!
Oh nvm, I think it's documented there.

@rickyyx rickyyx removed the @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. label Jan 6, 2023
krfricke
krfricke approved these changes Jan 6, 2023
View reviewed changes
Copy link
Contributor

@krfricke krfricke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

richardliaw
richardliaw requested changes Jan 6, 2023
View reviewed changes
doc/source/ray-overview/installation.rst Outdated
`Linux Python 3.8`_ `MacOS Python 3.8`_ `Windows Python 3.8`_
`Linux Python 3.7`_ `MacOS Python 3.7`_ `Windows Python 3.7`_
`Linux Python 3.6`_ `MacOS Python 3.6`_
`Linux Python 3.11`_ `MacOS Python 3.10`_ `Windows Python 3.10`_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we avoid this "off by one" format? maybe just put 3.11 at the bottom (and label as experimental if necessary?)

@csko
update docs
241dc32 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
richardliaw
richardliaw previously requested changes Jan 6, 2023
View reviewed changes
Copy link
Contributor

@richardliaw richardliaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ricky, the table doesn't seem to render anymore

@csko
fix table in documentation
d72e1d7 
Signed-off-by: Kornel Csernai <749306+csko@users.noreply.github.com>
pcmoritz
pcmoritz approved these changes Jan 7, 2023
View reviewed changes
Copy link
Contributor

@pcmoritz pcmoritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, but I did notice that the download link for the Python 3.11 wheels in the docs is currently not working. Will that be automatically fixed once the PR is merged or do we need to do anything else?

@pcmoritz pcmoritz dismissed richardliaw’s stale review January 7, 2023 17:58

This has been fixed now

@pcmoritz
Copy link
Contributor

pcmoritz commented Jan 7, 2023

I checked that wheels are only uploaded after the PR is merged -- let's try to merge it :)

@pcmoritz pcmoritz merged commit 0c8b59d into ray-project:master Jan 7, 2023
@pcmoritz
Copy link
Contributor

pcmoritz commented Jan 7, 2023 •
edited

After merging and the master build finished, the link is now indeed working 🎉

AmeerHajAli pushed a commit that referenced this pull request Jan 12, 2023
@csko @AmeerHajAli
Prepare Python 3.11 support (#31170)
4ddc7c3 
Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues.

### Main changes

- Upgrade cython to 0.29.32
- Add CI/CD steps for 3.11 wheel
- Change cython code to not use `recursion_depth`
- Update cloudpickle to latest
- Use newer manylinux2014 which has python3.11
- Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version

### Checklist:

- cython
  - [x] remove deprecated `recursion_depth`
  - [x] exc_type cython/cython#4500
- [ ] package dependencies https://pyreadiness.org/3.11/
  - [ ] llvmlite numba/llvmlite#869
  - [ ] numba numba/numba#8304
  - [ ] pyarrow
  - [ ] scikit-learn
  - [ ] pydantic
  - [x] cloudpickle
  - [x] upgrade cython to 0.29.32
  - [ ] tensorflow tensorflow/tensorflow#58032
  - [ ] torch pytorch/pytorch#86566
  - [ ] miniconda conda/conda#11170
    - [ ] `docker/base-deps/Dockerfile`
- [x] claim to support 3.11 in setup.py
- [ ] cicd
  - [ ] .buildkite/
    - [ ] .buildkite/pipeline.build.yml
  - [ ] ci/
    - [ ] ci/build/test-wheels.sh
    - [ ] ci/build/build-docker-images.py
  - [ ] release tests
  - [ ] docker/retag-lambda/python_versions.txt
  - [ ] download_wheels.sh
  - [ ] wheels
    - [ ] `python/build-wheel-macos.sh`
    - [ ] `python/build-wheel-windows.sh`
- [ ] Tests
  - [ ] pytest ray/serve/tests
  - [ ] python python/ray/serve/examples/echo_full.py
  - [ ] bazel test //:core_worker_test
  - [ ] bazel test --build_tests_only //:all
  - [ ] //python/ray/tests:test_pydantic_serialization tiangolo/fastapi#5048
  - [ ] //python/ray/train:test_torch_utils
- [ ] Documentation
  - [x] installation.rst

Current status: 
Linux and mac wheels build in CICD. Docker images will come in a separate PR.
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 16, 2023
@csko @tamohannes
Prepare Python 3.11 support (ray-project#31170)
8732258 
Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues.

### Main changes

- Upgrade cython to 0.29.32
- Add CI/CD steps for 3.11 wheel
- Change cython code to not use `recursion_depth`
- Update cloudpickle to latest
- Use newer manylinux2014 which has python3.11
- Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version

### Checklist:

- cython
  - [x] remove deprecated `recursion_depth`
  - [x] exc_type cython/cython#4500
- [ ] package dependencies https://pyreadiness.org/3.11/
  - [ ] llvmlite numba/llvmlite#869
  - [ ] numba numba/numba#8304
  - [ ] pyarrow
  - [ ] scikit-learn
  - [ ] pydantic
  - [x] cloudpickle
  - [x] upgrade cython to 0.29.32
  - [ ] tensorflow tensorflow/tensorflow#58032
  - [ ] torch pytorch/pytorch#86566
  - [ ] miniconda conda/conda#11170
    - [ ] `docker/base-deps/Dockerfile`
- [x] claim to support 3.11 in setup.py
- [ ] cicd
  - [ ] .buildkite/
    - [ ] .buildkite/pipeline.build.yml
  - [ ] ci/
    - [ ] ci/build/test-wheels.sh
    - [ ] ci/build/build-docker-images.py
  - [ ] release tests
  - [ ] docker/retag-lambda/python_versions.txt
  - [ ] download_wheels.sh
  - [ ] wheels
    - [ ] `python/build-wheel-macos.sh`
    - [ ] `python/build-wheel-windows.sh`
- [ ] Tests
  - [ ] pytest ray/serve/tests
  - [ ] python python/ray/serve/examples/echo_full.py
  - [ ] bazel test //:core_worker_test
  - [ ] bazel test --build_tests_only //:all
  - [ ] //python/ray/tests:test_pydantic_serialization tiangolo/fastapi#5048
  - [ ] //python/ray/train:test_torch_utils
- [ ] Documentation
  - [x] installation.rst

Current status: 
Linux and mac wheels build in CICD. Docker images will come in a separate PR.

Signed-off-by: tmynn <hovhannes.tamoyan@gmail.com>
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023
@csko @tamohannes
Prepare Python 3.11 support (ray-project#31170)
11a2027 
Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues.

### Main changes

- Upgrade cython to 0.29.32
- Add CI/CD steps for 3.11 wheel
- Change cython code to not use `recursion_depth`
- Update cloudpickle to latest
- Use newer manylinux2014 which has python3.11
- Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version

### Checklist:

- cython
  - [x] remove deprecated `recursion_depth`
  - [x] exc_type cython/cython#4500
- [ ] package dependencies https://pyreadiness.org/3.11/
  - [ ] llvmlite numba/llvmlite#869
  - [ ] numba numba/numba#8304
  - [ ] pyarrow
  - [ ] scikit-learn
  - [ ] pydantic
  - [x] cloudpickle
  - [x] upgrade cython to 0.29.32
  - [ ] tensorflow tensorflow/tensorflow#58032
  - [ ] torch pytorch/pytorch#86566
  - [ ] miniconda conda/conda#11170
    - [ ] `docker/base-deps/Dockerfile`
- [x] claim to support 3.11 in setup.py
- [ ] cicd
  - [ ] .buildkite/
    - [ ] .buildkite/pipeline.build.yml
  - [ ] ci/
    - [ ] ci/build/test-wheels.sh
    - [ ] ci/build/build-docker-images.py
  - [ ] release tests
  - [ ] docker/retag-lambda/python_versions.txt
  - [ ] download_wheels.sh
  - [ ] wheels
    - [ ] `python/build-wheel-macos.sh`
    - [ ] `python/build-wheel-windows.sh`
- [ ] Tests
  - [ ] pytest ray/serve/tests
  - [ ] python python/ray/serve/examples/echo_full.py
  - [ ] bazel test //:core_worker_test
  - [ ] bazel test --build_tests_only //:all
  - [ ] //python/ray/tests:test_pydantic_serialization tiangolo/fastapi#5048
  - [ ] //python/ray/train:test_torch_utils
- [ ] Documentation
  - [x] installation.rst

Current status: 
Linux and mac wheels build in CICD. Docker images will come in a separate PR.

Signed-off-by: tmynn <hovhannes.tamoyan@gmail.com>
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023
@csko @tamohannes
Prepare Python 3.11 support (ray-project#31170)
94f27ec 
Preliminary PR for adding Python 3.11 support, mapping out various dependencies, fixing issues.

### Main changes

- Upgrade cython to 0.29.32
- Add CI/CD steps for 3.11 wheel
- Change cython code to not use `recursion_depth`
- Update cloudpickle to latest
- Use newer manylinux2014 which has python3.11
- Condition certain python packages in requirements.txt on <3.11 that don't yet have a 3.11 version

### Checklist:

- cython
  - [x] remove deprecated `recursion_depth`
  - [x] exc_type cython/cython#4500
- [ ] package dependencies https://pyreadiness.org/3.11/
  - [ ] llvmlite numba/llvmlite#869
  - [ ] numba numba/numba#8304
  - [ ] pyarrow
  - [ ] scikit-learn
  - [ ] pydantic
  - [x] cloudpickle
  - [x] upgrade cython to 0.29.32
  - [ ] tensorflow tensorflow/tensorflow#58032
  - [ ] torch pytorch/pytorch#86566
  - [ ] miniconda conda/conda#11170
    - [ ] `docker/base-deps/Dockerfile`
- [x] claim to support 3.11 in setup.py
- [ ] cicd
  - [ ] .buildkite/
    - [ ] .buildkite/pipeline.build.yml
  - [ ] ci/
    - [ ] ci/build/test-wheels.sh
    - [ ] ci/build/build-docker-images.py
  - [ ] release tests
  - [ ] docker/retag-lambda/python_versions.txt
  - [ ] download_wheels.sh
  - [ ] wheels
    - [ ] `python/build-wheel-macos.sh`
    - [ ] `python/build-wheel-windows.sh`
- [ ] Tests
  - [ ] pytest ray/serve/tests
  - [ ] python python/ray/serve/examples/echo_full.py
  - [ ] bazel test //:core_worker_test
  - [ ] bazel test --build_tests_only //:all
  - [ ] //python/ray/tests:test_pydantic_serialization tiangolo/fastapi#5048
  - [ ] //python/ray/train:test_torch_utils
- [ ] Documentation
  - [x] installation.rst

Current status: 
Linux and mac wheels build in CICD. Docker images will come in a separate PR.

Signed-off-by: tmynn <hovhannes.tamoyan@gmail.com>
@grizzlybearg
Copy link

Are there any plans for windows python 3.11 wheels?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pcmoritz pcmoritz pcmoritz approved these changes

@wuisawesome wuisawesome wuisawesome approved these changes

@richardliaw richardliaw richardliaw approved these changes

@rickyyx rickyyx rickyyx approved these changes

@krfricke krfricke krfricke approved these changes

@amogkam amogkam Awaiting requested review from amogkam

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@edoakes edoakes Awaiting requested review from edoakes edoakes is a code owner

@DmitriGekhtman DmitriGekhtman Awaiting requested review from DmitriGekhtman

Assignees

@pcmoritz pcmoritz

@scv119 scv119

@rickyyx rickyyx

@krfricke krfricke

@simon-mo simon-mo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@csko @rickyyx @pcmoritz @grizzlybearg @wuisawesome @richardliaw @krfricke @scv119 @simon-mo