configure.py crashes connected to distutils and/or setuptools

[dwpaley]

Recent commits are hitting test failures for xfel_psana_centos_7 python3.9 on step "Configure and build" with tracebacks that look like this:

Traceback

Traceback (most recent call last):
  File "../modules/cctbx_project/libtbx/configure.py", line 34, in <module>
    if not run():
  File "../modules/cctbx_project/libtbx/configure.py", line 29, in run
    libtbx.env_config.cold_start(sys.argv)
  File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 3017, in cold_start
    env.refresh()
  File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 2251, in refresh
    module.process_libtbx_refresh_py()
  File "/__w/1/modules/cctbx_project/libtbx/env_config.py", line 2568, in process_libtbx_refresh_py
    exec(to_str(fh.read()), global_vars)
  File "<string>", line 24, in <module>
  File "/__w/1/modules/cctbx_project/libtbx/pkg_utils.py", line 54, in <module>
    import setuptools
  File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 73, in do_override
    ensure_local_distutils()
  File "/__w/1/miniforge/envs/psana_env/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /__w/1/miniforge/envs/psana_env/lib/python3.7/distutils/core.py

It looks like the same issue discussed in pypa/pip#10742 and pypa/setuptools#2993. I don't have any real idea what's going on, so just flagging this for now.

[bkpoon]

For now, set the version of setuptools to <60. That's what I did for the 2021.12 release and the nightly package builds.

[dwpaley]

For now, set the version of setuptools to <60. That's what I did for the 2021.12 release and the nightly package builds.

Thanks!

[phyy-nx]

Bump. Hit this again at NERSC. We are pinning the NERSC environments to setuptools<60 as well.

[Trzs]

Bump. Encountered this issue on a new Nersc test system. Pinning setuptools<60 solved it.

[dwpaley]

Alright, this is clearly a headache and is not fixing itself. I’ll look at it next week and try to figure out what’s going on.

[dwpaley]

Got some clarity on this from @abravalheri. It's apparently well known that importing pip will break setuptools because setuptools packages a modified distutils while pip requires the unmodified stdlib distutils. We can avoid using pip and setuptools at the same time, or else export an environment variable SETUPTOOLS_USE_DISTUTILS=stdlib.

[dwpaley]

On first glance I only see that pip is really needed in pkg_utils.require. @bkpoon any chance we can quarantine calls to require inside subprocesses so that we don't have to import pip otherwise?

[graeme-winter]

Bump. Encountered this issue on a new Nersc test system. Pinning setuptools<60 solved it.

In DIALS we got bitten by this and yes

silver-surfer-2 .conda-envs :) [main] $ grep setuptools *
linux.txt:conda-forge::setuptools<60
macos.txt:conda-forge::setuptools<60
windows.txt:conda-forge::setuptools<60

We just pinned setuptools