distutils and setuptools._distutils stubs are different

[sobolevn]

As far as I understand, setuptools._distutils is just a vendored copy of distutils.
But, right now they are different. For example:

• https://github.com/python/typeshed/blob/c1a857a41a1cbc62df5c6fdac52d0ac79fdf2824/stdlib/distutils/core.pyi
• https://github.com/python/typeshed/tree/c1a857a41a1cbc62df5c6fdac52d0ac79fdf2824/stubs/setuptools/setuptools/_distutils (no core.pyi)

Here's the source code for core.py:

• https://github.com/python/cpython/blob/3.11/Lib/distutils/core.py
• https://github.com/pypa/setuptools/blob/b545fc778583f644d6c331773dbe0ea53bfa41af/setuptools/_distutils/core.py

So, we need to work on them!
Refs #10249

[AlexWaygood]

Note that we deliberately removed much of our setuptools/_distutils stubs in #9795, on the grounds that setuptools appeared to view the directory as an implementation detail; it was a significant maintenance burden to keep our stubs for the directory up to date with the runtime package; and users should probably be importing things from setuptools instead of setuptools._distutils wherever possible.

FWIW I'm open to revisiting that decision if there's a good reason to, but we should think carefully about it. Prior to that change, it felt like stubtest was breaking every month due to the rate of changes setuptools was making to setuptools._distutils.

[hauntsaninja]

My opinions:

stdlib distutils is gone. Most users already do not get it when they import distutils. I have no interest in seeing further improvements made to its stubs.

setuptools._distutils seems not infrequently used in the wild, and I imagine it will be used more by projects looking to support 3.12 with the fewest changes possible. Given this, I'm supportive of improvements to these stubs.

Regarding maintenance burden from setuptools development of setuptools._distutils: I default to the position that users are worse in the world where we don't have stubs at all (certain false positive) than in the world where stubs exist + we stubtest allowlist them (possible incorrect behaviour, but likely only if depending on implementation detail)

Of course, if there's no usage of the stubs then better off staying deleted, e.g. I don't see much point in bringing back the setuptools._distutils.command.* removed in #9795

So all of this adds up to the following possible plan of action:

• Never touch stdlib distutils again, unless some user specifically asks, until 2028 when rm -rf.
• Improve setuptools._distutils, specifically focussing on the stub files that already exist
• Try vaguely to avoid trying back things deleted in Remove most of setuptools._distutils #9795, unless needed by existing stubs or used by users (for instance, we should probably bring back setuptools._distutils.ccompiler)
• If additions cause maintenance burden on setuptools updates, we can allowlist more aggressively. We can also just let the stubsabot upgrade PR languish if we don't have time for it (e.g. the stripe one has been sitting around for most of a year and I'm fine with that)
• And I don't view any of this as high priority, I think fine to do other stuff and wait for user issues

(re maintenance: I think part of the problem setuptools has is that they themselves are not sure what is implementation detail or not. Leaving typing aside, I've been broken by setuptools a dozen times in the last couple years)

[AlexWaygood]

That all sounds good; happy to cosign @hauntsaninja's plan of action (...in particular the last bullet 😉)

[JukkaL]

It looks like distutils is importable if setuptools is installed, even on Python 3.12:

$ python3
Python 3.12.0b3 (main, Jul  3 2023, 17:56:25) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> distutils.__file__
'/home/jukka/venv/test/lib/python3.12/site-packages/setuptools/_distutils/__init__.py'

I wonder if distutils isn't going away after all, and maybe setuptools stubs should ship with distutils stubs that just expose setuptools._distutils?

[abravalheri]

stdlib distutils is gone. Most users already do not get it when they import distutils. I have no interest in seeing further improvements made to its stubs.

It looks like distutils is importable if setuptools is installed, even on Python 3.12

As noted by the lead setuptools developer in pypa/setuptools#2806 (comment) and pypa/setuptools#2806 (comment), setuptools does allow users importing distutils directly via a MetaPathFinder. For all typing effects setuptools provides the implementation of distutils even for Python 3.12+ (with a "deprecated" status).

setuptools._distutils seems not infrequently used in the wild, and I imagine it will be used more by projects looking to support 3.12 with the fewest changes possible. Given this, I'm supportive of improvements to these stubs.

(re maintenance: I think part of the problem setuptools has is that they themselves are not sure what is implementation detail or not.

setuptools._distutils is merely an implementation detail and should not be imported directly. Importing it directly causes well-known runtime errors (e.g. pypa/setuptools#3743). The only supported way of importing stuff from distutils by setuptools is via import distutils/from distutils... statements.

If typeshed wants to support the stubs for the distribution of distutils that gets installed together with setuptools, that would need to be done via the distutils name not setuptools._distutils.

[Avasam]

I know we could ship distutils-stubs as part of types-setuptools. The main issue would still be the amount of code duplication and maintainability. (and whether all of typeshed's tooling would recognize it correctly)
We could try something with partial once the typeshed-internal/stub_uploader#109 fix is merged.
Or re-importing everything from stdlib distutils with star imports everywhere.
Either way only manually typing/updating any issue raised by stubtest.

---

Personally I'd prefer putting the efforts into typing setuptools itself and reducing false-positives (even if it means more false-negatives) in typeshed for distutils users in the short-to-mid term.