[BUG] setuptools 58.5 breaks installation of docutils==0.10.0

[anntzer]

setuptools version

58.5.{0,1}

Python version

3.9

OS

Arch Linux

Additional environment information

No response

Description

Installing docutils==0.10 in a clean venv with setuptools 58.5.{0,1} fails with an internal(?) AttributeError.

Expected behavior

Successful installation.

How to Reproduce

python -mvenv /tmp/testenv && source /tmp/testenv/bin/activate &&
  pip install -U pip setuptools wheel && pip install docutils==0.10

(To repro the issue, you may need to pip cache purge first if you happen to already have a wheel for docutils==0.10 in your pip cache.)

Output

  ERROR: Command errored out with exit status 1:
   command: /tmp/testenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_qz_txff
       cwd: /tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/
  Complete output (35 lines):
  running egg_info
  creating /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info
  writing /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-_qz_txff/docutils.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py", line 221, in <module>
      do_setup()
    File "/tmp/pip-install-r8b4xqgi/docutils_459783958fce4136b27e3bc6ec961dba/setup.py", line 98, in do_setup
      dist = setup(**kwargs)
    File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/usr/lib/python3.9/distutils/command/sdist.py", line 226, in add_defaults
      self._add_defaults_python()
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/sdist.py", line 113, in _add_defaults_python
      self._add_data_files(self._safe_data_files(build_py))
    File "/tmp/testenv/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 621, in _safe_data_files
      return build_py.get_data_files_without_manifest()
    File "/usr/lib/python3.9/distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: get_data_files_without_manifest
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/66/40/2b11ec2ec96d527ef933332a5847cc90860fd8bbee46da2cc468794ea25b/docutils-0.10.tar.gz#sha256=370624e61b6773da2f2fb17cc2a4eaea4bb596c3585d13f75ff193c1c738603e (from https://pypi.org/simple/docutils/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement docutils==0.10 (from versions: 0.3, 0.3.5, 0.3.7, 0.3.9, 0.4, 0.5, 0.6, 0.7, 0.8, 0.8.1, 0.9, 0.9.1, 0.10, 0.11, 0.12, 0.13.1, 0.14rc1, 0.14rc2, 0.14, 0.15, 0.15.1.post1, 0.15.2, 0.16b0.dev0, 0.16rc1, 0.16, 0.17b1, 0.17, 0.17.1b1.dev0, 0.17.1, 0.18b1, 0.18)
ERROR: No matching distribution found for docutils==0.10

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jerryz1982]

breaks pycrypto too

[jiasli]

Breaks python setup.py bdist_wheel -d build in our pipeline:

Full call stack

2021-11-04T03:23:41.8710919Z Traceback (most recent call last):
2021-11-04T03:23:41.8711856Z   File "/home/vsts/work/1/s/src/azure-cli/setup.py", line 160, in <module>
2021-11-04T03:23:41.8712337Z     setup(
2021-11-04T03:23:41.8713101Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/__init__.py", line 159, in setup
2021-11-04T03:23:41.8713849Z     return distutils.core.setup(**attrs)
2021-11-04T03:23:41.8714381Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/core.py", line 148, in setup
2021-11-04T03:23:42.0792338Z     dist.run_commands()
2021-11-04T03:23:42.0793141Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 966, in run_commands
2021-11-04T03:23:42.1846501Z     self.run_command(cmd)
2021-11-04T03:23:42.1847638Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.1848465Z     cmd_obj.run()
2021-11-04T03:23:42.1849964Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 335, in run
2021-11-04T03:23:42.1850889Z     self.run_command('install')
2021-11-04T03:23:42.1851397Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2316182Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2317514Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2320510Z     cmd_obj.run()
2021-11-04T03:23:42.2321832Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
2021-11-04T03:23:42.2322583Z     return orig.install.run(self)
2021-11-04T03:23:42.2323717Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/command/install.py", line 558, in run
2021-11-04T03:23:42.2335935Z     self.run_command(cmd_name)
2021-11-04T03:23:42.2336648Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2337157Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2337636Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2338089Z     cmd_obj.run()
2021-11-04T03:23:42.2349505Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/install_egg_info.py", line 34, in run
2021-11-04T03:23:42.2350417Z     self.run_command('egg_info')
2021-11-04T03:23:42.2350922Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 313, in run_command
2021-11-04T03:23:42.2351403Z     self.distribution.run_command(command)
2021-11-04T03:23:42.2351893Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/dist.py", line 985, in run_command
2021-11-04T03:23:42.2357369Z     cmd_obj.run()
2021-11-04T03:23:42.2358817Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 299, in run
2021-11-04T03:23:42.2359947Z     self.find_sources()
2021-11-04T03:23:42.2361214Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
2021-11-04T03:23:42.2362230Z     mm.run()
2021-11-04T03:23:42.2363085Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 541, in run
2021-11-04T03:23:42.2364084Z     self.add_defaults()
2021-11-04T03:23:42.2364807Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
2021-11-04T03:23:42.2365253Z     sdist.add_defaults(self)
2021-11-04T03:23:42.2365673Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/command/sdist.py", line 226, in add_defaults
2021-11-04T03:23:42.2366081Z     self._add_defaults_python()
2021-11-04T03:23:42.2366796Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/sdist.py", line 113, in _add_defaults_python
2021-11-04T03:23:42.2367678Z     self._add_data_files(self._safe_data_files(build_py))

2021-11-04T03:23:42.2368540Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 621, in _safe_data_files
2021-11-04T03:23:42.2369747Z     return build_py.get_data_files_without_manifest()
2021-11-04T03:23:42.2370277Z   File "/opt/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/distutils/cmd.py", line 103, in __getattr__
2021-11-04T03:23:42.2370751Z     raise AttributeError(attr)
2021-11-04T03:23:42.2374899Z AttributeError: get_data_files_without_manifest

get_data_files_without_manifest was added by #2844

[abravalheri]

This seems to be related to the replacement of 8a0e63a with b6ce536.

Those entry-points should point to the setuptools version of the commands, but for some reason, the original distutils commands are being used instead.

---

Building the project with build seems to work fine, though:

$ virtualenv venv-docutils
$ source venv-docutils/bin/activate
$ pip cache purge
$ svn checkout https://svn.code.sf.net/p/docutils/code/trunk/docutils docutils-code
$ cd docutils-code
$ pip install build
$ python -m build

[nshgraph]

This appears to be a breaking change introduced in 2e66eb7 when custom commands are used as it forces the new function get_data_files_without_manifest to be present.

[jjerphan]

This appears to be a breaking change introduced in 2e66eb7 when custom commands are used as it forces the new function get_data_files_without_manifest to be present.

I confirm that pip installing using 1812f8d works but using 2e66eb7 fails (tested on scikit-learn).

[Alphare]

Also breaks install for Mercurial: https://foss.heptapod.net/mercurial/mercurial-devel/-/jobs/260776#L53

[abravalheri]

I think we have 2 separated problems here:

1. If packages define their own build_py inheriting from distutils instead of setuptools, and don't define a get_data_files_without_manifest, there will be a problem.

2. Installing sdist packages with pip might result in error, since the distutils version of build_py is being used instead of the setuptools version.
   This can be reproduced with docutils==0.10.0.
   Interestingly, pypa/build isolated environments don't seem to have the same problem.
   This second issue seems more difficult to track down.

---

Correction, this is indeed a single problem, as pointed out by @nshgraph

docutils==0.10.0 defines a custom version of copy_build_py_2to3 that inherits from distutils.command.build_py.build_py_2to3.

The latest version of docutils do not have this requirement. @anntzer are you unable to update the version of docutils?

[anntzer]

I certainly can, this was just a version I had pinned in CI to test an "oldest-supported-version-of-dependency" for another package. I guess the other commenters here may want a more robust solution, though...

[abravalheri]

I certainly can, this was just a version I had pinned in CI to test an "oldest-supported-version-of-dependency" for another package. I guess the other commenters here may want a more robust solution, though...

Yes, I understand that.

I am having a look to try to identify what can be done:

• Some packages listed here use build_py_2to3 (e.g. docutils, pycrypto)
• Others define a custom build_py inheriting directly from distutils (e.g. pycryptodome)
• There is a third case that I am still trying to find a connection. This is the case of packages that redefine build_ext, sdist or custom builds but not build_py.

---

@jaraco maybe the easiest approach is to restore 8a0e63a?

Unless supporting commands that inherit directly from distutils is outside of the scope of setuptools, which in that case would require adding a "breaking change" notice to the change log.

[jaraco]

I'll review this tonight and get out a fix or yank the release. Thanks for your patience through this unintentional disruption.