[BUG] release v65.6.0 breaks packages downstream due to removal of distutils.log.Log

[sphuber]

setuptools version

65.6.0

Python version

Python 3.10

OS

Ubuntu

Additional environment information

No response

Description

The distutils.log.Log class was removed in 74652ca. This causes popular packages downstream, such as numpy, to fail: see numpy/numpy#22623

Expected behavior

The module distutils.log module was not officially marked as deprecated even though 74652ca added to the docstring that the module is Retained for compatibility and should not be used.. It would be great if the removed class could be reinstated and a deprecation pathway be provided.

How to Reproduce

1. pip install setuptools==65.6.0
2. from numpy.distutils import Log

Output

In [4]: from numpy.distutils import Log
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-f8e71815afcd> in <module>
----> 1 from numpy.distutils import Log

~/.virtualenvs/aiida_dev/lib/python3.9/site-packages/numpy/distutils/__init__.py in <module>
     22 # Must import local ccompiler ASAP in order to get
     23 # customized CCompiler.spawn effective.
---> 24 from . import ccompiler
     25 from . import unixccompiler
     26 

~/.virtualenvs/aiida_dev/lib/python3.9/site-packages/numpy/distutils/ccompiler.py in <module>
     18 from distutils.version import LooseVersion
     19 
---> 20 from numpy.distutils import log
     21 from numpy.distutils.exec_command import (
     22     filepath_from_subprocess_output, forward_bytes_to_stdout

~/.virtualenvs/aiida_dev/lib/python3.9/site-packages/numpy/distutils/log.py in <module>
      2 import sys
      3 from distutils.log import *  # noqa: F403
----> 4 from distutils.log import Log as old_Log
      5 from distutils.log import _global_log
      6 

ImportError: cannot import name 'Log' from 'distutils.log' (/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/setuptools/_distutils/log.py)

[oscarbenjamin]

This actually breaks just importing numpy.distutils altogether:

In [2]: import numpy.distutils
---------------------------------------------------------------------------
ImportError
...
~/38venv/lib/python3.8/site-packages/numpy/distutils/log.py in <module>
      2 import sys
      3 from distutils.log import *  # noqa: F403
----> 4 from distutils.log import Log as old_Log
      5 from distutils.log import _global_log
      6 

ImportError: cannot import name 'Log' from 'distutils.log' (~/38venv/lib/python3.8/site-packages/setuptools/_distutils/log.py)

[bentheiii]

This is a serious issue for people using older numpy (that can only install with distutils) and poetry (as poetry doesn't respect PEP-518 AFAICT). We have no way to install our dependencies except to manually install numpy with the --no-build-isolation flag.

[maxnoe]

Also running into this, numpy 1.23.5 is also affected.

[cliffwoolley]

export SETUPTOOLS_USE_DISTUTILS=stdlib appears to be another workaround, which allowed me then to build/install scipy==1.4.1 on ARM (where there are no prebuilt binaries).

@bentheiii 's alternative above of pip install --no-build-isolation scipy==1.4.1 after pinning back setuptools was the only other way I got to work so far.

[bentheiii]

Note that this only appears in the latest poetry, downgrading to 1.1.15 appears to fix the issue

[yhosoai-coupa]

@bentheiii I'm running into this from numpy 1.19.4 also. what do I need to downgrade 1.1.15 to?
we are supposed to be on setuptools==53.0.0 but still running into this somehow started recently..

[tulbureandreit]

Running into this for statsmodels==0.13.0 and numpy==1.22.0

[cliffwoolley]

For numpy, follow one of the two options I listed for scipy. (Scipy hits it because of its dependence on numpy.)

[oscarbenjamin]

• Initially setuptools integration tests included numpy, however since numpy is pinning the setuptools version used, this integration test also became impractical and was removed.

I'm not sure I understand this. SymPy's CI just does pip install numpy and then tries to run a setup.py to build an extension module. I would have thought it would be straight-forward to have a CI job for setuptools that does the equivalent of that.

If you (or anyone else in the community) is interested in keeping the compatibility between numpy and setuptools

I am not personally interested in doing this. Both distutils and numpy.distutils are deprecated so the expected downstream response as I understand it is just to stop using them. I find it strange though that backwards incompatible changes would be made if these are deprecated and basically shouldn't be used any more.

The problem right now is that it isn't clear what the expected alternatives for distutils/numpy.distutils are supposed to be. Note that the SymPy code in question is using numpy.distutils in the way that is still suggested in the NumPy documentation:
https://numpy.org/doc/stable/user/c-info.ufunc-tutorial.html
The issue I already referred to (numpy/numpy#22027) discusses changing that and the discussion there found a solution that (probably) works for SymPy. That solution doesn't work for all projects though and I think that those docs are not updated just because it isn't clear what they should now be updated to say.

[cliffwoolley]

Another way out from this situation would have been pip constraints, but as far as I can tell, constraints don't seem to apply to an isolated build environment:

root@20ccdca7a737:/# echo 'setuptools<60' > constraints.txt
root@20ccdca7a737:/# pip install -c constraints.txt scipy==1.4.1 --verbose
Using pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
Collecting scipy==1.4.1
  Downloading scipy-1.4.1.tar.gz (24.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.6/24.6 MB 10.3 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Ignoring numpy: markers 'python_version == "3.5" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.5" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
  Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
  Collecting wheel
    Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
  Collecting setuptools
    Using cached setuptools-65.6.0-py3-none-any.whl (1.2 MB)  <---------------
  Collecting Cython>=0.29.13
    Downloading Cython-0.29.32-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (1.8 MB)

It looks like pypa/pip#10669 asked for exactly this (for the same reason of distutils breakage), but that one was duped onto pypa/pip#4582 which is still open.

[skirpichev]

On Tue, Nov 22, 2022 at 04:55:58AM -0800, Oscar Benjamin wrote: I think it is however easy to fix this in SymPy's case by switching to setuptools instead of numpy.distutils because I don't think the additional features of numpy.distutils are needed there.

I doubt. https://numpy.org/devdocs/reference/distutils_status_migration.html#moving-to-setuptools says that the setuptools has no Fortran build support, for example.

[tulbureandreit]

@tulbureandreit

I have a package which we use in Dev which is used with Poetry. Hence I cannot use your PIP-based fix.

One of the comments on the related numpy issue suggests that the env var approach does work with poetry. I haven't tried it myself, but intuitively it does seem like that option should be independent of pip. numpy/numpy#22623 (comment)

@cliffwoolley I cannot run it. We have dockerized the model and each time it does poetry install, it needs to install statsmodels, which statsmodels has no PIN for setuptools < 65.6.0, so it fails.

We pinned our own models constraints to be <65.6.0

Another Q would be, starting which version of numpy would we get rid of this problem? 1.23.1?

[jaraco]

I've cut a release 65.6.2 with the fix. Sorry it took so long to get a remedy out. That was mainly my fault. Huge thanks to abravalheri for multiple fixes and validation work.

[cliffwoolley]

Thanks @jaraco @abravalheri !

(FWIW @tulbureandreit , I'm also building Docker images, though using pip rather than poetry. Setting the env var with ENV is what I'm doing so that it applies to any and all pip installs.)

[cliffwoolley]

For those diving right in to verify like I am, it seems 65.6.1 also has the relevant change; 65.6.2 that @jaraco mentioned was about half an hour later with relnote-only tweaks. v65.6.1...v65.6.2 . 65.6.2 hasn't made it to PyPI yet as of this writing https://pypi.org/project/setuptools/#history , but presumably would be there soon. EDIT: 65.6.2 is also now on PyPI.

[oscarbenjamin]

Thanks all.

[tulbureandreit]

@cliffwoolley I solved it with SETUPTOOLS_USE_DISTUTILS=stdlib poetry install. I heard maybe 65.6.2 fixed it ?

[haraldschilly]

I can confirm that setuptools version 65.6.3 fixes this, i.e. I got that numpy error when importing pyscf.

[eli-schwartz]

• Initially setuptools integration tests included numpy, however since numpy is pinning the setuptools version used, this integration test also became impractical and was removed.

There should probably be a distinction between building numpy (with pinned setuptools) in CI, versus building a carefully crafted test case that depends on numpy and setuptools, and does from numpy.distutils import ....

The latter is what really needs to be tested here in order to make sure that yet more projects using numpy.distutils are operational.