pip installation fails with the most recent setuptools (v65.6.0)

[pacesm]

The pip spacepy installation fails with the following error:

File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/numpy/distutils/log.py", line 6, in <module>
          from distutils.log import Log as old_Log
      ImportError: cannot import name 'Log' from 'distutils.log' (/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/_distutils/log.py)

The error is caused by the most recent setuptools v65.6.0 release (pypa/setuptools#3693) and it is related to the distutils and numpy.distutils deprecation.

As a quick remedy, setting

SETUPTOOLS_USE_DISTUTILS=stdlib

before calling the pip installation helps.

Required fixes, as suggested by the NumPy documentation:

1. Pin setuptools to a version < 60 in the package requirements as a quick fix.
2. Get rid of the deprecated numpy.distutils in the long run.

---

Environment: clean Ubuntu 20.04 (GitHub CI), Python >= 3.7

Installation steps (failing at the spacepy step):

apt-get install gfortran
apt-get install libhdf5-dev
( cd libcdf && make build && sudo make install )
pip3 install --upgrade pip
pip3 install wheel
pip3 install scipy
pip3 install spacepy

Full error dump:

Collecting spacepy
  Downloading spacepy-0.4.1.zip (18.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 81.0 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy>=0.11 in /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages (from spacepy) (1.7.3)
Requirement already satisfied: numpy!=1.15.0,>=1.10 in /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages (from spacepy) (1.21.6)
Collecting python-dateutil>=1.4
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 52.7 MB/s eta 0:00:00
Collecting matplotlib>=1.5
  Downloading matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 122.9 MB/s eta 0:00:00
Collecting h5py>=2.6
  Downloading h5py-3.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 132.6 MB/s eta 0:00:00
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 36.1 MB/s eta 0:00:00
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 104.4 MB/s eta 0:00:00
Collecting pillow>=6.2.0
  Downloading Pillow-9.3.0-cp37-cp37m-manylinux_2_28_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 130.3 MB/s eta 0:00:00
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting packaging>=20.0
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 14.5 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
  Downloading fonttools-4.38.0-py3-none-any.whl (965 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 965.4/965.4 kB 113.2 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting typing-extensions
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Building wheels for collected packages: spacepy
  Building wheel for spacepy (pyproject.toml): started
  Building wheel for spacepy (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for spacepy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 250, in build_wheel
          metadata_directory)
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 413, in build_wheel
          wheel_directory, config_settings)
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 484, in run_setup
          self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 40, in <module>
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/numpy/distutils/__init__.py", line 8, in <module>
          from . import ccompiler
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/numpy/distutils/ccompiler.py", line 17, in <module>
          from numpy.distutils import log
        File "/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/numpy/distutils/log.py", line 6, in <module>
          from distutils.log import Log as old_Log
      ImportError: cannot import name 'Log' from 'distutils.log' (/tmp/pip-build-env-jjzhjlyn/overlay/lib/python3.7/site-packages/setuptools/_distutils/log.py)
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for spacepy
ERROR: Could not build wheels for spacepy, which is required to install pyproject.toml-based projects
Failed to build spacepy

[jtniehof]

Thanks for the workaround. We knew this was coming but it bit sooner than expected. I'm in the process of moving the installer to a pure setuptools based approach, which will address this.

[pacesm]

Update: The broken setuptools have been fixed in the meantime and I can confirm that the previously broken installation works again. The underlying distutils problem remains though.

I leave it at your discretion whether to close the ticket or keep it open.

[jtniehof]

Thanks. I'm going to leave this open just so it's apparent to people and will close once we're no longer using numpy distutils.

[jtniehof]

0.5.0 no longer uses distutils directly, and #725 captures the problem of the f2py interface changing without distutils. Closing.