Adopt bdist_wheel from pypa/wheel

[abravalheri]

Summary of changes

pypa/wheel maintainers previously described that the long term vision is to transfer bdist_wheel directly to setuptools.
This PR attempts to do that based on the previous changes (that untangled the way Setuptools produced core-metadata and dist-info from pypa/wheel).
It also relies on the WheelBuilder as introduced in a previous PR (the pypa/wheel APIs for building wheel files are likely to change in the future).

---

This is part of a series of PRs:

• Avoid modification of install_requires and extra_requires that deviates from core metadata #3903
• Improve core metadata compliance for PKG-INFO #3904
• Avoid undocumented pypa/wheel API in dist_info #3905
• Implement a simplified wheel builder (avoid undocumented pypa/wheel API) #3906
• Avoid pypa/wheel API in editable_wheel #3907
• Adopt bdist_wheel from pypa/wheel #3908

The motivation for this series of PRs is the following:

• Logic for generating .egg-info and .dist-info directories is intertwined and implicit
  (See Remove dependency on wheel #1386).
• Setuptools uses pypa/wheel API which is not stable yet and is very likely to change in the future.
• pypa/wheel maintainers previously described that the long term vision is to transfer bdist_wheel
  directly to setuptools (See pypa/wheel#262, pypa/wheel#472, pypa/wheel#472).

---

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[abravalheri]

Hi @agronholm, after reading some previous comments about bdist_wheel I assumed that there is an overall long term intent to move bdist_wheel to the setuptools project. That is what motivated this PR.

I had a look on the pypa/wheel codebase and noticed that macosx_libfile seems to only be used to support bdist_wheel. Am I correct in assuming that macosx_libfile would also be moved to setuptools together with bdist_wheel?

[agronholm]

The macosx_libfile module is used to extract architectural information from macos dynamic libraries in order to determine the appropriate tags for binary wheels. It could conceivably be needed outside of setuptools, if other packaging back-ends start using the new public API.

[abravalheri]

Thank you very much for the clarification @agronholm.

We might do the transition in 2 steps then... In the first step we bundle macosx_libfile inside setuptools. Then in a second step, we can use it directly from wheel when it is exposed via a public API. Does that sound good?

[agronholm]

Yeah, you'd have to vendor all the necessary modules from wheel in the first step. I think we'd need to coordinate this so as to minimize any duplication between setuptools and wheel. What's the intended time table for this?

[abravalheri]

I don't have a timeline for this yet. It is a complex change, so I tried to split it up in many smaller steps (e.g., fixing PKG-INFO, removing uses of private pypa/wheel APIs, etc ...). I am hopping that the other maintainers/members of the community will have sometime to have a look on the proposed PRs and give some feedback at some point.

Maybe on the second semester when I have a bit more time, I might start pushing a little more.
What I had in mind is:

1. Collect feedback about the proposed changes and implement all related modifications.
2. Publish a beta release and ask for beta testers on the Packaging topic in the Python discourse.
3. Implement eventual fixes
4. Create the final release (major version bump).

[abravalheri]

The main idea was to get setuptools out of the way for pypa/wheel to advance the works on the public API without risking breaking existing packages. Then later come back and check for opportunities to use this public API and simplify setuptools code base.

[agronholm]

I have been wondering if it makes any sense for all the PyPA stuff to be split between so many different projects. Maybe the wheel related code should be moved to packaging instead?

[abravalheri]

I have nothing against this change... For example, it seems that the contents macosx_libfile and other functions to calculate tags could be useful there.

If the packaging maintainers are happy to absorb some functionality that would also be a good alternative¹.

Footnotes

1. I am assuming that bdist_wheel would still be part of setuptools, since it makes more sense. ↩

[agronholm]

Yeah, I think it would be nice if the new public api for wheel would be implemented in packaging instead, and wheel itself was reduced to nothing more than a command line tool. bdist_wheel needs to be migrated to setuptools in any event. The only reason the code still lives in the wheel package is of course historical.

I think I'll open an issue with packaging to discuss this. Then there's a question of whether auditwheel and wheel should be merged together, but I think that's a separate issue.

[agronholm]

pypa/packaging#697