Build failure with 2.2.0 #892

Cadair · 2021-10-26T15:17:06Z

Description

Hello,

I am in the middle of updating some builds and they all started failing as the 2.2.0 release is now being installed over the 2.1 version.

The error I am seeing is:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.7/x64/bin/cibuildwheel", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 138, in main
    options = compute_options(platform=platform, command_line_arguments=args)
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/options.py", line 564, in compute_options
    options = Options(platform=platform, command_line_arguments=command_line_arguments)
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/options.py", line 328, in __init__
    self.reader = OptionsReader(
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/options.py", line 174, in __init__
    self.default_options, self.default_platform_options = self._load_file(defaults_path)
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/options.py", line 243, in _load_file
    with filename.open("rb") as f:
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/cibuildwheel/resources/defaults.toml'

Build log

https://dev.azure.com/astropy-project/astropy/_build/results?buildId=2128&view=logs&j=77547fd3-4b07-56ce-e8f3-7c919931387b&t=bc3eb0f8-3b1a-57cf-720b-e49aeb7434de&l=31

CI config

No response

The text was updated successfully, but these errors were encountered:

pypa/cibuildwheel#892

Czaki · 2021-10-26T15:37:24Z

the file is missed from wheel and sdist. @joerick please yank release.

henryiii · 2021-10-26T15:45:52Z

Are we not using check-manifest? :'(

joerick · 2021-10-26T15:49:45Z

I don't have time to investigate this now, but I've yanked the release. Very strange considering that the beta didn't have any issues. Will take a look later on.

henryiii · 2021-10-26T15:52:51Z

It's fine from the GHA, it's only a problem if you use the PyPI package. I'd expect most of the beta testing is from the action.

How are we producing the SDist and wheel? I don't see an upload job, etc? I also don't see a MANIFEST.in? I'm a strong believer in always uploading SDists and wheels from CI. (Well, obviously, I'm in a project called cibuildwheel....) Too easy to make mistakes locally.

henryiii · 2021-10-26T15:54:44Z

@joerick if you can generate a token and upload it, I can add the CI job for making releases (will look just like https://scikit-hep.org/developer/gha_pure )

I can add a MANIFEST.in and check-manifest job.

henryiii · 2021-10-26T16:24:22Z

Yes, it looks like SDists have always been incomplete. The default SDist inclusion does not include py.typed or *.toml. It does include *.cfg and *.txt. So my guess is @joerick has been running this locally, and updated to a >=0.5.0 version of build, which builds SDist -> wheel, rather than directly building wheel. Since the SDists have always been broken, this caused the wheel to now also be broken.

pip install --build-from-source cibuildwheel likely has not worked since at least 2.0. I checked, the 2.1.3 SDist definitely is broken.

joerick · 2021-10-26T17:19:15Z

Yes, I just shifted to using python -m build for the last release. I suppose the wheel never had this issue before.

Happy to look at a CI-based release - looks like that is in #894, thanks

henryiii · 2021-10-26T17:35:50Z

python -m build —wheel would avoid the SDist, and so would have included this file. But the SDists have been broken and needed to be fixed, which is why Build defaults to SDist -> wheel.

joerick · 2021-10-26T18:10:33Z

I've just uploaded 2.2.1b1 using the CI-based approach. Seems to have worked. I'll double-check that the PyPI assets are good and then do a proper release of 2.2.1.

henryiii · 2021-10-26T18:20:19Z

$ unzip -l cibuildwheel-2.2.1b1-py3-none-any.whl
Archive:  cibuildwheel-2.2.1b1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
       24  10-26-2021 18:06   cibuildwheel/__init__.py
     9611  10-26-2021 18:06   cibuildwheel/__main__.py
     3933  10-26-2021 18:06   cibuildwheel/architecture.py
     3804  10-26-2021 18:06   cibuildwheel/bashlex_eval.py
     7842  10-26-2021 18:06   cibuildwheel/docker_container.py
     3119  10-26-2021 18:06   cibuildwheel/environment.py
     1009  10-26-2021 18:06   cibuildwheel/extra.py
    15418  10-26-2021 18:06   cibuildwheel/linux.py
     8793  10-26-2021 18:06   cibuildwheel/logger.py
    23312  10-26-2021 18:06   cibuildwheel/macos.py
    21479  10-26-2021 18:06   cibuildwheel/options.py
     2355  10-26-2021 18:06   cibuildwheel/projectfiles.py
        0  10-26-2021 18:06   cibuildwheel/py.typed
      980  10-26-2021 18:06   cibuildwheel/typing.py
    10253  10-26-2021 18:06   cibuildwheel/util.py
    16164  10-26-2021 18:06   cibuildwheel/windows.py
     7758  10-26-2021 18:06   cibuildwheel/resources/build-platforms.toml
     1068  10-26-2021 18:06   cibuildwheel/resources/constraints-python36.txt
      984  10-26-2021 18:06   cibuildwheel/resources/constraints-python37.txt
      811  10-26-2021 18:06   cibuildwheel/resources/constraints-python38.txt
      811  10-26-2021 18:06   cibuildwheel/resources/constraints-python39.txt
       41  10-26-2021 18:06   cibuildwheel/resources/constraints.in
      811  10-26-2021 18:06   cibuildwheel/resources/constraints.txt
     1129  10-26-2021 18:06   cibuildwheel/resources/defaults.toml
     1434  10-26-2021 18:06   cibuildwheel/resources/install_certifi.py
     1949  10-26-2021 18:06   cibuildwheel/resources/pinned_docker_images.cfg
     1374  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/LICENSE
    22059  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/METADATA
       92  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/WHEEL
       61  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/entry_points.txt
       13  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/top_level.txt
     2814  10-26-2021 18:06   cibuildwheel-2.2.1b1.dist-info/RECORD
---------                     -------
   171305                     32 files

$ tar -tf cibuildwheel-2.2.1b1.tar.gz
cibuildwheel-2.2.1b1/
cibuildwheel-2.2.1b1/LICENSE
cibuildwheel-2.2.1b1/MANIFEST.in
cibuildwheel-2.2.1b1/PKG-INFO
cibuildwheel-2.2.1b1/README.md
cibuildwheel-2.2.1b1/cibuildwheel/
cibuildwheel-2.2.1b1/cibuildwheel/__init__.py
cibuildwheel-2.2.1b1/cibuildwheel/__main__.py
cibuildwheel-2.2.1b1/cibuildwheel/architecture.py
cibuildwheel-2.2.1b1/cibuildwheel/bashlex_eval.py
cibuildwheel-2.2.1b1/cibuildwheel/docker_container.py
cibuildwheel-2.2.1b1/cibuildwheel/environment.py
cibuildwheel-2.2.1b1/cibuildwheel/extra.py
cibuildwheel-2.2.1b1/cibuildwheel/linux.py
cibuildwheel-2.2.1b1/cibuildwheel/logger.py
cibuildwheel-2.2.1b1/cibuildwheel/macos.py
cibuildwheel-2.2.1b1/cibuildwheel/options.py
cibuildwheel-2.2.1b1/cibuildwheel/projectfiles.py
cibuildwheel-2.2.1b1/cibuildwheel/py.typed
cibuildwheel-2.2.1b1/cibuildwheel/resources/
cibuildwheel-2.2.1b1/cibuildwheel/resources/build-platforms.toml
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints-python36.txt
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints-python37.txt
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints-python38.txt
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints-python39.txt
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints.in
cibuildwheel-2.2.1b1/cibuildwheel/resources/constraints.txt
cibuildwheel-2.2.1b1/cibuildwheel/resources/defaults.toml
cibuildwheel-2.2.1b1/cibuildwheel/resources/install_certifi.py
cibuildwheel-2.2.1b1/cibuildwheel/resources/pinned_docker_images.cfg
cibuildwheel-2.2.1b1/cibuildwheel/typing.py
cibuildwheel-2.2.1b1/cibuildwheel/util.py
cibuildwheel-2.2.1b1/cibuildwheel/windows.py
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/PKG-INFO
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/SOURCES.txt
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/dependency_links.txt
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/entry_points.txt
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/not-zip-safe
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/requires.txt
cibuildwheel-2.2.1b1/cibuildwheel.egg-info/top_level.txt
cibuildwheel-2.2.1b1/pyproject.toml
cibuildwheel-2.2.1b1/setup.cfg
cibuildwheel-2.2.1b1/setup.py
cibuildwheel-2.2.1b1/test/
cibuildwheel-2.2.1b1/test/test_0_basic.py
cibuildwheel-2.2.1b1/test/test_before_all.py
cibuildwheel-2.2.1b1/test/test_before_build.py
cibuildwheel-2.2.1b1/test/test_before_test.py
cibuildwheel-2.2.1b1/test/test_build_skip.py
cibuildwheel-2.2.1b1/test/test_cpp_standards.py
cibuildwheel-2.2.1b1/test/test_dependency_versions.py
cibuildwheel-2.2.1b1/test/test_docker_images.py
cibuildwheel-2.2.1b1/test/test_emulation.py
cibuildwheel-2.2.1b1/test/test_environment.py
cibuildwheel-2.2.1b1/test/test_macos_archs.py
cibuildwheel-2.2.1b1/test/test_manylinuxXXXX_only.py
cibuildwheel-2.2.1b1/test/test_pep518.py
cibuildwheel-2.2.1b1/test/test_pure_wheel.py
cibuildwheel-2.2.1b1/test/test_ssl.py
cibuildwheel-2.2.1b1/test/test_subdir_package.py
cibuildwheel-2.2.1b1/test/test_testing.py
cibuildwheel-2.2.1b1/test/test_troubleshooting.py
cibuildwheel-2.2.1b1/test/test_wheel_tag.py

Looks okay to me.

henryiii · 2021-10-26T18:21:16Z

Though it's a little odd to include test but not unit_test in the SDist, there's no harm in it.

henryiii · 2021-10-26T18:22:35Z

It's in the default rules: https://packaging.python.org/guides/using-manifest-in/#how-files-are-included-in-an-sdist. The default rules are weird, as always.

joerick · 2021-10-26T18:28:32Z

v2.2.1 is live on PyPI! Thank you to @Cadair for reporting and @Czaki and @henryiii for debugging :)

Though it's a little odd to include test but not unit_test in the SDist, there's no harm in it.

It's in the default rules: https://packaging.python.org/guides/using-manifest-in/#how-files-are-included-in-an-sdist. The default rules are weird, as always.

Good old setuptools! Yeah that's a crazy rule. I don't think there's much point in including tests in sdists really, we have publicly accessible git repos now!

henryiii · 2021-10-26T19:21:41Z

I personally like having tests in SDists; it's much easier in some cases (Conda recipes & system package managers, at least). Unless they are large, there's not really a cost (and pretty much the only people downloading SDists are probably the type who might want tests). I don't think either of these really applies to cibuildwheel, though.

I'm totally fine to go either way, but we should be consistent, and provide the whole test suite or no test suite. If we only had one, the unit test suite would be more useful. :)

Cadair added a commit to OpenAstronomy/azure-pipelines-templates that referenced this issue Oct 26, 2021

Exclude cibuildwheel 2.2.0 as it seems to be broken

e7768fe

pypa/cibuildwheel#892

This was referenced Oct 26, 2021

fix: produce working SDists, check in CI #893

Merged

ci: release to PyPI from CI on GitHub Release #894

Merged

henryiii mentioned this issue Oct 26, 2021

no v2.2.0 release on pypi yet? #891

Closed

joerick closed this as completed Oct 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failure with 2.2.0 #892

Build failure with 2.2.0 #892

Cadair commented Oct 26, 2021

Czaki commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021 •

edited

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021 •

edited

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021 •

edited

Build failure with 2.2.0 #892

Build failure with 2.2.0 #892

Comments

Cadair commented Oct 26, 2021

Description

Build log

CI config

Czaki commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021 • edited

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021 • edited

henryiii commented Oct 26, 2021

henryiii commented Oct 26, 2021

joerick commented Oct 26, 2021

henryiii commented Oct 26, 2021 • edited

henryiii commented Oct 26, 2021 •

edited

henryiii commented Oct 26, 2021 •

edited

henryiii commented Oct 26, 2021 •

edited