Python 3.11 wheel for macOS does not work for x86_64

[jbms]

What did you do?

Attempted to use cibuildwheel to build a package that depends on Pillow.

  ImportError while loading conftest '/Users/runner/work/neuroglancer/neuroglancer/python/tests/conftest.py'.
  work/neuroglancer/neuroglancer/python/tests/conftest.py:16: in <module>
      import neuroglancer.webdriver
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/neuroglancer/__init__.py:16: in <module>
      from .server import set_static_content_source, set_server_bind_address, is_server_running, stop
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/neuroglancer/server.py:31: in <module>
      from . import local_volume, static
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/neuroglancer/local_volume.py:29: in <module>
      from .chunks import encode_jpeg, encode_npz, encode_raw
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/neuroglancer/chunks.py:19: in <module>
      from PIL import Image
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/PIL/Image.py:100: in <module>
      from . import _imaging as core
  E   ImportError: dlopen(/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/PIL/_imaging.cpython-311-darwin.so, 2): no suitable image found.  Did find:
  E   	/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-95w581j7/cp311-macosx_x86_64/venv-test/lib/python3.11/site-packages/PIL/_imaging.cpython-311-darwin.so: mach-o, but wrong architecture

What are your OS, Python and Pillow versions?

• OS: macOS 11.6.8 (github actions macos-latest)
• Python: 3.11
• Pillow: 9.2.0

[nulano]

From your build log, this seems to be using a universal wheel, Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2.whl.

However, using a file analyzer, I get:

>trid.exe Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2\PIL\_imaging.cpython-311-darwin.so

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  14992
Analyzing...

Collecting data from file: Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2\PIL\_imaging.cpython-311-darwin.so
100.0% (.) Mac OS X Mach-O 64-bit ARM executable (8000/1)

I'm also not aware of any intent to release universal wheels, and there are 2 universal wheels, both for Python 3.11 on https://pypi.org/project/Pillow/9.2.0/.

I suspect this is actually an M1 wheel with an incorrect tag.

The other "universal" wheel appears to be x86_64:

>trid.exe Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2\PIL\_imaging.cpython-311-darwin.so

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  14992
Analyzing...

Collecting data from file: Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2\PIL\_imaging.cpython-311-darwin.so
100.0% (.) Mac OS X Mach-O 64-bit Intel executable (8000/1)

Does installing this other wheel, Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2.whl, before running tests work?

[jbms]

I'm not sure how easily I can modify my build to test that, but per your analysis it looks pretty clear that the wheels are incorrectly tagged/named.

Are you able to remove them and re-generate them with the correct tags?

[nulano]

I've tried to reproduce using GitHub Actions and the wheel Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2.whl I linked above does seem to work: https://github.com/nulano/Pillow/runs/8123829414?check_suite_focus=true#step:6:2022

However, Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2.whl is refusing to install because appears to macosx_11_0 not be supported by the version of Python on GitHub Actions: https://github.com/nulano/Pillow/runs/8123829338?check_suite_focus=true#step:6:2042

AFAIK PyPI doesn't allow removing wheels, and build tags (which have been used here to fix broken builds in the past) have a lower priority than platform tags, so I'm not sure what is the best way to replace these.

Perhaps reuploading the x86_64 wheel as Pillow-9.2.0-cp311-cp311-macosx_10_10_x86_64.whl as well as Pillow-9.2.0-cp311-cp311-macosx_11_0_x86_64.whl might work?

FWIW, the wheels were built with Python 3.11b3, and I'm getting some segfaults if I upgrade setuptools, so it might be better to rebuild a new wheel with Python 3.11rc1. Related: python-pillow/Pillow#6367

[jbms]

PyPI does allow you to delete individual files from the web interface.

[radarhere]

I've created PR #317 to fix the generated wheels. Here is a wheel that it created -
Pillow-9.2.0-cp311-cp311-macosx_11_0_arm64.whl.zip

PyPI does allow you to delete individual files from the web interface.

This doesn't match my experience when I login to PyPI. I suspect we meet some of the exceptions outlined in https://discuss.python.org/t/stop-allowing-deleting-things-from-pypi/17227

The final version of Python 3.11 is due to be released on October 3rd. On October 15th, we are due to release the next version of Pillow with support for that. Is it possible to just wait until then to upload a new wheel?

[jbms]

I guess you could contact the pypi admins to delete the file. If the other wheels are uploaded, will they take precedence?

In any case I have already worked around the problem for my package by skipping running tests for Python 3.11 for now.

[hugovk]

PR #317 is merged, thanks!

PyPI does allow you to delete individual files from the web interface.

This doesn't match my experience when I login to PyPI.

That's because you and I only have a "maintainer" role, not the "owner" role that can delete:

When I check another PyPI project where I'm an owner, there's options to delete files (or releases):

We can ask Alex or Eric to delete files.

On October 15th, we are due to release the next version of Pillow with support for that. Is it possible to just wait until then to upload a new wheel?

It would be good to have working 3.11 wheels ahead of the Python 3.11 final release to help people test and potentially find bugs in Python itself during the RC phase.

Should we upload one or both of the new wheels?

• Pillow-9.2.0-cp311-cp311-macosx_10_10_x86_64.whl
• Pillow-9.2.0-cp311-cp311-macosx_11_0_arm64.whl

And delete one or both of the old ones?

• Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2.whl
• Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2.whl

[radarhere]

If the other wheels are uploaded, will they take precedence?

Yes. So we don't need to delete original wheels, we can just upload new ones. We've actually done this before, for Pillow 9.0.1.

[hugovk]

Sounds good, let's upload those with 3.11 RC1: #318

[radarhere]

I've uploaded the two new Python 3.11 wheels to PyPI, one for x86_64 and one for arm64.

However, I realise now that my statement about the new wheels taking precedence was based on a false idea. I was imagining that we were uploading wheels with the same names as the old ones.
Apparently "pip will prefer a native wheel over a universal2 one" though, and testing on my M1, the arm64 wheel is downloaded.
Except that testing on my Intel, the universal2 wheel is still used.

So I guess to resolve this properly, we will need @aclark4life or wiredfool to delete the Pillow-9.2.0-cp311-cp311-macosx_11_0_universal2.whl and Pillow-9.2.0-cp311-cp311-macosx_10_10_universal2.whl files from PyPI.

[aclark4life]

@radarhere Done 👍

[radarhere]

Thanks very much!

@jbms this should now be fixed. Did you want to confirm?

[jbms]

Thanks, yes this is fixed now.