Add support for Python 3.11

[pmeier]

As of today, Python 3.11 is not available in the defaults channel of Anaconda, which we are building against: https://anaconda.org/anaconda/python

As stated in #6841 (comment), we cannot build against the conda-forge channel.

As soon as it is available in the defaults channel, there is #6959 to add support in our CI and in turn for the nightlies and the next release.

[CtrlC-Root]

ContinuumIO/anaconda-issues#13082

[johnnynunez]

conda create --name pytorch python=3.11 -c conda-forge -y
Torchvision is blocking my tests for upgrade to pytorch 1.13.1 with 3.11

[pmeier]

@johnnynunez You are using the conda-forge channel. As explained above, we are not building against that, but rather the defaults channel:

$ conda create --name pytorch python=3.11 --dry-run
[...]
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.11

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
[...]

[johnnynunez]

@johnnynunez You are using the conda-forge channel. As explained above, we are not building against that, but rather the defaults channel:

$ conda create --name pytorch python=3.11 --dry-run
[...]
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.11

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
[...]

They are working on that: conda/conda#11170
Why does it rely on conda and not on python binaries as in pytorch repository?

[pmeier]

Why does it rely on conda and not on python binaries as in pytorch repository?

Because we are not just building against the Python ABI, but a few other libraries as well. Mainly libpng and libjpeg.

[johnnynunez]

Why does it rely on conda and not on python binaries as in pytorch repository?

Because we are not just building against the Python ABI, but a few other libraries as well. Mainly libpng and libjpeg.

Thank you for clarify

[malfet]

I'm curious, if we should just add python 3.11 library to pytorch channel to unblock the wheels builds.

[pmeier]

@malfet I favor this. If we do, we should also consider 4. of #6563 since we are missing out on quite a big performance improvement again due to lack of support in the defaults channel.

[johnnynunez]

@pmeier Python 3.11 is out in anaconda
ContinuumIO/anaconda-issues#13082 (comment)

conda create -n py311 python=3.11

[pentium10]

any progress on this one? pytorch was enabled for linux builds, and wondering if vision is up to a point to be tested?

[pmeier]

@pentium10 As explained above, we need more than just python=3.11 to be available. See #7108 (comment). Until all of that is available as well, we can't provide 3.11 builds.

[pentium10]

numpy, Pillow, scipy, requests, pytest, setuptools all have release builds for python3.11
what do I miss?

[pmeier]

@pentium10 You miss that they have no builds against Python 3.11 on the conda defaults channel. And that is what we are building against. So we need Anaconda to build these packages for 3.11 now that they recently added support for it. But there is nothing that we can do on that front other than ask.

[malfet]

@pmeier I agree we could not publish conda package until dependencies are not available on Anaconda, but I somewhat disagree that we should hold of wheel builds due to that.
But to unblock some of the pipelines I'm going to build and publish some of the dependencies to pytorch-nightly channel (already published pytest, numpy, scipy and setuptools for Linux, can easily add pillow there)

[powderluv]

yes can we please have 3.11 torchvision wheels

[cornking03]

Why does it rely on conda and not on python binaries as in pytorch repository?

Because we are not just building against the Python ABI, but a few other libraries as well. Mainly libpng and libjpeg.

Is there a document/post explaining (in terms that someone who does not develop libraries can understand) the reliance on Anaconda for the CI process? The default channel may require a paid license, so I was just curious why that was chosen.

Are there any alternatives to Anaconda for CI? This seems to be one of the big things hindering progress. I'm not trying to suggest that retooling a CI process would be trivial - just trying to understand the design decisions.

[malfet]

yes can we please have 3.11 torchvision wheels

I think we do have them now.

% pip install torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu/
% python -c "import sys,torch,torchvision;print(sys.version, torch.__version__, torchvision.__version__)"
3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:26:40) [Clang 14.0.6 ] 2.0.0.dev20230209 0.15.0.dev20230209

[powderluv]

just validated it works in downstream torch-mlir CI llvm/torch-mlir#1848

Thank you.

[Erotemic]

Any update for 3.11 torchvision on pypi?

[pmeier]

@Erotemic Release is planned for March 15.