Support building pure Python wheels

[pombredanne]

As a follow up to #1007 I would like to also allow building a pure wheel.
Why? When I build arbitrary wheels, I cannot know ahead of time if a wheel will be pure Python or native. And if I build many wheels including sdist tarballs as in #597 (comment) there is no way I can know ahead of time if a built wheel will be pure python or not.

[Czaki]

Why? When I build arbitrary wheels, I cannot know ahead of time if a wheel will be pure Python or native. And if I build many wheels including sdist tarballs as in #597 (comment) there is no way I can know ahead of time if a built wheel will be pure python or not.

But to build native wheels you need to spin up 3 different operating systems. For pure wheel, you need only one OS.

short inspection of setup.py will inform if there is any compiled code. Or simply run a local build of the wheel and check the output file.

Building a pure wheel from a non-pure python package is an error that happens (misconfiguration) and pointing a user that you produce pure wheel is part of error reporting.

And If you got such an error you could simply switch to call of build.

[pombredanne]

short inspection of setup.py will inform if there is any compiled code. Or simply run a local build of the wheel and check the output file.

that's easy for one, hard for 100, impossible for a 1000.

[pombredanne]

I pushed a minimal implementation with a test. It works with a new --pure-wheel option and keeps on trucking rather than failing in this case. See #1022

[Czaki]

short inspection of setup.py will inform if there is any compiled code. Or simply run a local build of the wheel and check the output file.

that's easy for one, hard for 100, impossible for a 1000.

Writle simple shell script which will call tar -xvf sdist (or equivalent for zip). Then python -m build package_dir and check if there is wheele in dist (if not then there is need of manual intervention) and if there is system name in wheel (linux, win, macosx depending on host system) and you will got information if you need to use cibuildwheel.

And if You have such script then you could even process 10 000 packages

I pushed a minimal implementation with a test. It works with a new --pure-wheel option and keeps on trucking rather than failing in this case

To use such flag you need first determine if package is pure python. What is advantage of using cibuildwheel?

[pombredanne]

@Czaki I appreciate the feedback and I know how to drive this alright ... See https://github.com/nexB/skeleton/tree/main/etc/scripts if you are interested
That said and FWIW you cannot determine if a wheel will be pure before you build it. Just building it and discaring it as it was done until now in cibuildwheel does not make sense. I am just keeping the built wheel optionally.

To use such flag you need first determine if package is pure python. What is advantage of using cibuildwheel?

Not at all. cibuildwheel today ALWAYS build a wheel, pure or not. And then raise an Exception if the wheel was pure.

At scale you cannot know if a wheel will be pure unless you build it first as a given build context could lead to build a plain wheel for some package that have optional native speedup extensions.

Now, are suggesting something like:

1. build it locally first. I already have code for this https://github.com/nexB/skeleton/blob/b1dabd894c469174b0fee950043f7d29fac6027f/etc/scripts/utils_thirdparty.py#L2829 which essentially does this already.
2. if pure, build it with cibuildwheel

What I want is something simpler: build it with cibuildwheel, and get the results out, pure or not. No need to guess.

[Czaki]

What I want is something simpler: build it with cibuildwheel, and get the results out, pure or not. No need to guess.

But one of the common errors of packages that have optional native dependency is to build the pure wheel, instead of the native one. So there is a need to raise such an exception.

So cibuildwheel cannot produce pure wheel by default, to not force package maintainers for inspecting the list of produced wheels on every release (which will consume much more time than setup).

Additionally to build a pure wheel one need only one runner, not three.

These are two arguments why building pure wheels need a different workflow than building a native one. So what is the argument for doing it using cibuildwheel instead of plain build call?

[pombredanne]

But one of the common errors of packages that have optional native dependency is to build the pure wheel, instead of the native one. So there is a need to raise such an exception.

and this is not changed in the PR. The pure wheel is kept IFF it exists and the --pure-wheel option was created.

Additionally to build a pure wheel one need only one runner, not three.

As I said, the stock cibuildwheel already build pure wheels across all the runners anyway. It just discards them and fail if they were created. My addition keeps them optionally.

These are two arguments why building pure wheels need a different workflow than building a native one. So what is the argument for doing it using cibuildwheel instead of plain build call?

Again, cibuildwheel already builds plain wheels. So your argument does not hold IMHO.

[pombredanne]

These are two arguments why building pure wheels need a different workflow than building a native one. So what is the argument for doing it using cibuildwheel instead of plain build call?

As I said above, I want to avoid having too many moving parts and I having to guess if a wheel is pure or not.

[joerick]

I don't fully understand the use-case for this yet. You'd like an automated way to build wheels for 100-1000 sdists? This kinda makes sense, except that, as @Czaki says:

But to build native wheels you need to spin up 3 different operating systems

So what would be the strategy here? Build on linux and if a platform wheel pops out, build on macos and windows as well? Or build on all three and arbitrarily pick one?

Cards on the table: I am still slightly biased against adding this feature, simply because build exists, which is a simpler tool which already does this. I suppose that it doesn't test the wheels against multiple versions of python, so there is some rationale for cibuildwheel. But we have to balance these user needs against the priorities of clarity of direction of the project, and maintenance/support burden. I also note that this has been discussed before: #255, #862.

[pombredanne]

@joerick

I don't fully understand the use-case for this yet. You'd like an automated way to build wheels for 100-1000 sdists?

I maintain Python tools like https://github.com/nexB/scancode-toolkit that depend on quite a wheel packages with a mix of pure and native code. I support linux windows and macos and provide self-contained installable archives that contain all the prebuilt binaries. This way the tool can be installed on Windows or macOS even if there are no build tools or toolchain on hand.
To achieve this I need prebuilt wheels of all the combos OS/Arch/Python versions (together with sdists) even if there is so such wheel built on PyPI by their maintainers. I I build all these wheels and them push them a public mini PyPI index to use either as an extra-index-url for pip or as the source for building installable packages for each combos OS/Arch/Python otherwise.

If I would get all these prebuilt upstream on PyPI, I would not need many of this!

This kinda makes sense, except that, as @Czaki says:

But to build native wheels you need to spin up 3 different operating systems

You are already buidling the plain wheels... rather I mean cibuildwheel already builds the plain wheels and then after than if it sees this is a plain/pure wheel it fails the whole build. I am for now just adding a simple option to not fail the build and still provide the pure wheel that was built. This will three time the wheel, so I will surely discard one and this could surely be streamlined... but I am not sure you can have a matrix job alert other in flight job not to build some wheel because it was a pure wheel that built in another matrix job ;)

Which one of the three I would pick does not matter much as they should be binary identical (or rather likely will be when we get reproducible built wheels)... at least the content and the same is the same.

Cards on the table: I am still slightly biased against adding this feature, simply because build exists, which is a simpler tool which already does this. I suppose that it doesn't test the wheels against multiple versions of python, so there is some rationale for cibuildwheel.

Testing is part of the appeal and also just taking advantage that the pure was built and will be always built in all cases.
The other appeal is having a gas plant-like machinery to hand off build to many different build loops and tools.

With all that said, if you do not want it that's OK: I will maintain my fork alright... but I think this is a disservice to cibuildwheels users to always fail on pure wheels and in earnest I would rather apply my energy to the greater good and helping furthering this tools than maintains my own more complicated loop.

[pombredanne]

@zanecodes FYI since this is related to #862

[Czaki]

I suppose that it doesn't test the wheels against multiple versions of python, so there is some rationale for cibuildwheel.

tox, nox - why additional tool?

I maintain Python tools like nexB/scancode-toolkit that depend on quite a wheel packages with a mix of pure and native code. I support linux windows and macos and provide self-contained installable archives that contain all the prebuilt binaries. This way the tool can be installed on Windows or macOS even if there are no build tools or toolchain on hand.
To achieve this I need prebuilt wheels of all the combos OS/Arch/Python versions (together with sdists) even if there is so such wheel built on PyPI by their maintainers. I I build all these wheels and them push them a public mini PyPI index to use either as an extra-index-url for pip or as the source for building installable packages for each combos OS/Arch/Python otherwise.

So you ask fo option to remove "--no-deps" from pip wheel command. Then in one call collect all wheels?

cibuildwheel/cibuildwheel/linux.py

Lines 196 to 208 in b52698d

	docker.call(
	[
	"python",
	"-m",
	"pip",
	"wheel",
	container_package_dir,
	f"--wheel-dir={built_wheel_dir}",
	"--no-deps",
	*verbosity_flags,
	],
	env=env,
	)

[henryiii]

I don't think this is reasonable or going to work for you. It's quite rare that a binary wheel will actually build without any dependencies. You have no idea of how to test an arbitrary wheel, so you might not even know if it wasn't going to work. If someone is very careful and has no binary dependencies, then the default absolutely-no-cibuildwheel-configuration might work; I'm pretty sure boost-histogram falls into this category, but it requires a lot of work to do that properly, PEP 518 builds, etc. And if a library knows enough about proper packaging to do this, they probably already provided wheels too.

cibuildwheel is designed to be configured. You have to tell it how to test your package. You have to tell it how to prepare for binary builds. There have been thoughts about standardizing this (see #564), and cibuildwheel now has toml config that could be picked up, but again, anyone with this would also be providing wheels.

For pure wheels, we provide pypa/build, and that's the right tool to use. For downloading, pip download. Assuming you build on multiple Python versions, you'd be still waisting a ton of CI time building each Python version and then just overwriting the same wheel.

that's easy for one, hard for 100, impossible for a 1000.

So you are building infrastructure to build a 1000 wheels at a time, but can't be bothered to put a tiny bit of logic in to save at least 3x and probably 12x of your time most of the time by using pip download, build, and then resorting to cibuildwheel? Logic that could easily sit in a noxfile? You are still going to handle the fact that 80% of the binary wheels won't build because they need extra setup, or are going to be produced broken/missing parts but you won't know because you don't know how to test them.

[pombredanne]

I suppose that it doesn't test the wheels against multiple versions of python, so there is some rationale for cibuildwheel.

tox, nox - why additional tool?

That's probably not what @joerick was referring to and that I was thinking about: cibuildwheel does a simple installation test of each built wheel in the OS/arch/Python context of the build to ensure this can install correctly. That's not in tox and nox job specs to do this IMHO, this is more a simple smoke test check at the tail end of teh build.

[pombredanne]

@Czaki re:

So you ask fo option to remove "--no-deps" from pip wheel command. Then in one call collect all wheels?

this could work out too... great idea.

[henryiii]

cibuildwheel does a simple installation test

No it does not. It does nothing at all by default. You can configure it to run your tests, and if you put in echo "hi", then it does a simple installation test. But it requires configuration.

That's not in tox and nox job specs

Nox doesn't have specs. You can do a parametrized Python job and install from your wheel in it. It's about 3 lines. Tox probably can do it too, but I tend to avoid tox, at least until tox 4 is out.

Parametrizing across OS's is done by your CI, not ?ox. Archs are a tiny bit harder, but you are likely paramatrizing across those too - and we added support for pipx in manylinux, so any manylinux arch can run pipx run nox and that will have all supported Pythons available.

I believe all the tools you need are available separately, and there's no need to try to force cibuildwheel to do something it's not designed to do.

[pombredanne]

@henryiii re:

cibuildwheel is designed to be configured.

At least for the code I help maintain like pyahocorasick or intbitset, these are self contained, so would be bitarray, these being some of the native dependencies that come to mind. I reckon that building larger machinery is involved alright, but that's not my target.

So you are building infrastructure to build a 1000 wheels at a time, but can't be bothered to put a tiny bit of logic in to save at least 3x and probably 12x of your time most of the time by using pip download, build, and then resorting to cibuildwheel? Logic that could easily sit in a noxfile?

I already have code for this alright: https://github.com/nexB/skeleton/blob/b1dabd894c469174b0fee950043f7d29fac6027f/etc/scripts/utils_thirdparty.py#L2829

You are still going to handle the fact that 80% of the binary wheels won't build because they need extra setup, or are going to be produced broken/missing parts but you won't know because you don't know how to test them.

That's not been my experience and in practice even if I may no know how to run the test suites of these third-party packages, I have extensive test suites in my packages that make this a lesser issue

[pombredanne]

now, if you guys think this is not a feature for here, I am fine to close this, I will do it elsewhere, no hard feelings!

[henryiii]

I think we should assign a custom error code to pure-wheel "failures", then I think it would still solve your case (except maybe for smoke-check tests, but those are easy to run manually, say with nox).

[joerick]

I think we should assign a custom error code to pure-wheel "failures"

This is a good idea. Then you could fall back to PyPA-build when the wheel is pure.

[henryiii]

Technically, the wheel is already built - we are checking the wheel tags via the name, so you could just use that. (Though it should be pretty fast, regardless, so rebuilding wouldn't be expensive)

[henryiii]

We should make a table in the docs and have a few return codes, ideally. This one could come with a note that the built wheel will be available in the output folder if it is returned.

[pombredanne]

For downloading, pip download.

@henryiii FWIW, pip download is fairly difficult to use in practice: either it fails or takes a fairly long time pypa/pip#10917

[pombredanne]

We should make a table in the docs and have a few return codes, ideally. This one could come with a note that the built wheel will be available in the output folder if it is returned.

@henryiii I think I finally understood ... did you mean to have a specific numeric error code returned by the command line?

[joerick]

Having a tidy up. Closing this as a non-goal of cibuildwheel.

[Marco-Sulla]

My personal experience: my module current lacks the support for Cpython 3.11. So for this Cpython version it produces a pure python wheel.

Some of the projects that uses my module use cibuildwheel to build and test it, as me. But they fails at 3.11.

Personally, I put CIBW_SKIP: "cp311-* pp*", but this is not what the majority of people does. I know that one project abandoned my module for this reason.

[Czaki]

But why people are building wheel on it's own instead using your wheel?

[henryiii]

You can specify in your pyproject.toml:

[tool.cibuildwheel]
skip=["cp311-*", "pp*"]

Then anyone will get this without setting environment variables. Though cibuildwheel is not replacement for nox/tox, or some other general tool for building and testing. It's a tool to make wheels for upload to PyPI. This should only be done once, not by "projects using modules"?

Also, make sure you are uploading a pure Python wheel too; installers pick the specific one from PyPI first, but fall back on the pure one if one is not available (will help other platforms, too, no one should be building SDists in that case unless they ask for them).

[Marco-Sulla]

But why people are building wheels on its own instead of using your wheels?

For the C extension, I don't really know X-D
About the pure Python wheel, read below.

You can specify in your pyproject.toml:

[tool.cibuildwheel]
skip=["cp311-*", "pp*"]

Thank you, I'll try it.

cibuildwheel is not replacement for nox/tox [...] It's a tool to make wheels for upload to PyPI

Well, but so why cibuildwheel support testing, if it's only a tool for making wheels? ;-)

Also, make sure you are uploading a pure Python wheel too

I used to do it, but people asked me to remove pure python wheels.

This is because they use a not supported arch. pip, instead of trying to build the C Extension from the sdist, falls back to the pure python package.

So I preferred to remove the pure python package and let the sdist to generate the pure python package.

[Marco-Sulla]

This is because they use a not supported arch. pip, instead of trying to build the C Extension from the sdist, falls back to the pure python package.

PS: about this I opened an issue on pip:
pypa/pip#11818

[Czaki]

Well, but so why cibuildwheel support testing, if it's only a tool for making wheels? ;-)

to validate if the wheels were build properly.

I used to do it, but people asked me to remove pure python wheels.

This is because they use a not supported arch. pip, instead of trying to build the C Extension from the sdist, falls back to the pure python package.

then maybe upload name-version-py311-none-any.whl to pypi. Then it will be only picked for python 3.11

[henryiii]

You can use pipx run "wheel>=0.40" tags --python-tag=py311 --remove <wheel_name> to convert a py3 wheel into a py311 wheel.

[henryiii]

I should also point out you have to opt-into the testing (with [tool.cibuildwheel] test-command = ...). It doesn't even by default test the wheels it builds, and there's no way to test a preexisting wheel.

Having pip try to build the SDist and then fall back to a pure Python wheel sounds like a bad idea. I think it's better to provide a pure Python implementation and then users who really need the performance can use pip's build from source flag. The cost to try to build the SDist will be high and likely produce a lot of output. If you cover a large fraction of users with the binaries, they should mostly be happy. Some users want reproducible environments and don't want to be building SDists, which is not reproducible. The recent failed PEP on locking environments didn't even allow SDist builds, for example.

[Marco-Sulla]

You can specify in your pyproject.toml:

[tool.cibuildwheel]
skip=["cp311-*", "pp*"]

I tested it and it works, but only if CIBW_SKIP is not specified. I hope this will never happen.

You can use pipx run "wheel>=0.40" tags --python-tag=py311 --remove <wheel_name> to convert a py3 wheel into a py311 wheel.

I get this error and Gugol does not help me:

(venv_3_11) marco@buzz:~/sources/python-frozendict$ pipx run "wheel>=0.40" tags --python-tag=py311 --remove frozendict
⚠  wheel is already on your PATH and installed at /home/marco/sources/python-frozendict/venv_3_11/bin/wheel. Downloading and
    running anyway.
'wheel' executable script not found in package 'wheel'.
Available executable scripts:
    
(venv_3_11) marco@buzz:~/sources/python-frozendict$

I should also point out you have to opt-into the testing (with [tool.cibuildwheel] test-command = ...). It doesn't even by default test the wheels it builds, and there's no way to test a preexisting wheel.

Not sure to had understood. Do you mean I have to add the tests by default? IMHO it should the user that have to decide to run the tests or not, since they are also a bit slow (they tests against segfaults and memory leaks too).

The recent failed PEP on locking environments didn't even allow SDist builds, for example.

Well, PEP 665 was rejected because the lack of sdist support indeed :)

[Czaki]

I get this error and Gugol does not help me:

I'm not sure if it is connected, but you need to provide the path to .whl file. Something like pipx run "wheel>=0.40" tags --python-tag=py311 --remove dist/tox_min_req-0.0.5.dev1+g2c33795-py3-none-any.whl

[henryiii]

Ahh, sorry, pipx has a bug with wheel. It filters wheel normally since it's part of the default venv environment and usually you don't want this executable to show up as a surprising extra when you are getting some other package. Someone is working on this bug as of a few days ago. For now, just install wheel however you install dependencies and then run wheel with the remainder of that command.

[Marco-Sulla]

@henryiii Thank you, it worked:

$ python -m wheel tags --python-tag=py311 --remove dist/frozendict-2.3.8-py3-none-any.whl 
frozendict-2.3.8-py311-none-any.whl
$

But does it only rename the wheel?

[henryiii]

It changes the wheel metadata to reflect the new tags too.

[cburroughs]

We should make a table in the docs and have a few return codes, ideally.

My reading of this thread is that there intended to be a specific exit code for NonPlatformWheelError, but I'm just seeing 1 in practice.