macOS python3.11 fails to install mdanalysis[all] from git branch directly (not always though)

[marinegor]

Expected behavior

I want to install full MDAnalysis package from dask-0 branch on macOS laptop.

Actual behavior

I get following log:

❯ uv pip install './package[all]'
error: Failed to download and build: mdanalysis @ file:///private/tmp/mdanalysis/package
  Caused by: Failed to build: mdanalysis @ file:///private/tmp/mdanalysis/package
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
--- stdout:
Attempting to autodetect OpenMP support... Did not detect OpenMP support.
No openmp compatible compiler found default to serial build.
Will attempt to use Cython.
Compiling MDAnalysis/lib/formats/libdcd.pyx because it changed.
Compiling MDAnalysis/lib/c_distances.pyx because it changed.
Compiling MDAnalysis/lib/c_distances_openmp.pyx because it changed.
Compiling MDAnalysis/lib/qcprot.pyx because it changed.
Compiling MDAnalysis/lib/formats/libmdaxdr.pyx because it changed.
Compiling MDAnalysis/lib/formats/cython_util.pyx because it changed.
Compiling MDAnalysis/analysis/encore/cutils.pyx because it changed.
Compiling MDAnalysis/analysis/encore/clustering/affinityprop.pyx because it changed.
Compiling MDAnalysis/analysis/encore/dimensionality_reduction/stochasticproxembed.pyx because it changed.
Compiling MDAnalysis/lib/_cutil.pyx because it changed.
Compiling MDAnalysis/lib/_augment.pyx because it changed.
Compiling MDAnalysis/lib/nsgrid.pyx because it changed.
Compiling MDAnalysis/coordinates/timestep.pyx because it changed.
[ 1/13] Cythonizing MDAnalysis/analysis/encore/clustering/affinityprop.pyx
[ 2/13] Cythonizing MDAnalysis/analysis/encore/cutils.pyx
[ 3/13] Cythonizing MDAnalysis/analysis/encore/dimensionality_reduction/stochasticproxembed.pyx
[ 4/13] Cythonizing MDAnalysis/coordinates/timestep.pyx
[ 5/13] Cythonizing MDAnalysis/lib/_augment.pyx
[ 6/13] Cythonizing MDAnalysis/lib/_cutil.pyx
[ 7/13] Cythonizing MDAnalysis/lib/c_distances.pyx
[ 8/13] Cythonizing MDAnalysis/lib/c_distances_openmp.pyx
[ 9/13] Cythonizing MDAnalysis/lib/formats/cython_util.pyx
--- stderr:
performance hint: MDAnalysis/lib/c_distances.pyx:301:5: Exception check on '_minimum_image_orthogonal' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '_minimum_image_orthogonal' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '_minimum_image_orthogonal' to allow an error code to be returned.
performance hint: MDAnalysis/lib/c_distances.pyx:328:5: Exception check on '_minimum_image_triclinic' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '_minimum_image_triclinic' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '_minimum_image_triclinic' to allow an error code to be returned.
performance hint: MDAnalysis/lib/c_distances.pyx:416:37: Exception check after calling '__pyx_fuse_0_minimum_image_orthogonal' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '__pyx_fuse_0_minimum_image_orthogonal' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '__pyx_fuse_0_minimum_image_orthogonal' to allow an error code to be returned.
performance hint: MDAnalysis/lib/c_distances.pyx:416:37: Exception check after calling '__pyx_fuse_1_minimum_image_orthogonal' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '__pyx_fuse_1_minimum_image_orthogonal' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '__pyx_fuse_1_minimum_image_orthogonal' to allow an error code to be returned.
performance hint: MDAnalysis/lib/c_distances.pyx:440:36: Exception check after calling '__pyx_fuse_0_minimum_image_triclinic' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '__pyx_fuse_0_minimum_image_triclinic' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '__pyx_fuse_0_minimum_image_triclinic' to allow an error code to be returned.
performance hint: MDAnalysis/lib/c_distances.pyx:440:36: Exception check after calling '__pyx_fuse_1_minimum_image_triclinic' will always require the GIL to be acquired.
Possible solutions:
	1. Declare '__pyx_fuse_1_minimum_image_triclinic' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on '__pyx_fuse_1_minimum_image_triclinic' to allow an error code to be returned.

Error compiling Cython file:
------------------------------------------------------------
...
    array_wrapper = ArrayWrapper()
    array_wrapper.set_data(<void*> data_ptr, <int*> &dim[0], dim.size, data_type)

    cdef np.ndarray ndarray = np.array(array_wrapper, copy=False)
    # Assign our object to the 'base' of the ndarray object
    ndarray.base = <PyObject*> array_wrapper
           ^
------------------------------------------------------------

MDAnalysis/lib/formats/cython_util.pyx:115:11: Assignment to a read-only property
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
    super().run_setup(setup_script=setup_script)
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 593, in <module>
  File "<string>", line 451, in extensions
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/Users/marinegor/Library/Caches/uv/.tmpBml4q3/.venv/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: MDAnalysis/lib/formats/cython_util.pyx
---

Code to reproduce the behavior

uv venv && source .venv/bin/activate
git clone git@github.com:marinegor/mdanalysis.git --single-branch --branch feature/dask-0
uv pip install 'mdanalysis/package[all]'

Current version of MDAnalysis

$ python3 -V
Python 3.11.6
$ # mdanalysis version
ab394a49e305549e1ead392aa85b9af7c897791e commit hash
$ uname -a
Darwin marinegor_macos 21.6.0 Darwin Kernel Version 21.6.0: Mon Feb 19 20:24:34 PST 2024; root:xnu-8020.240.18.707.4~1/RELEASE_X86_64 x86_64

I also noticed something weird -- after the first time I got the bug, I couldn't reproduce it, i.e. the code above worked when I tested it while writing the issue, so I'm not sure how to reproduce it anymore. Anyway, raising it, since there might be something

[orbeckst]

I ran into a similar(?) issue with pip install (but not pip install -e) (see Discord message):

I am trying to install the development version (actually, the dask-0 branch from the parallelization PR), following the instructions in the user guide https://userguide.mdanalysis.org/stable/contributing_code.html#creating-a-development-environment but creating a Python 3.11 environment (on macOS) with the packages from https://userguide.mdanalysis.org/stable/contributing_code.html#id1

When doing pip install it complains about absolute paths and I get this error:

$ python -m pip install .
Processing ~/MDAnalysis/mdanalysis/package
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

...
      adding license file 'AUTHORS'
      writing manifest file 'MDAnalysis.egg-info/SOURCES.txt'
      error: Error: setup script specifies an absolute path:

          ~/MDAnalysis/mdanalysis/package/MDAnalysis/analysis/encore/cutils.c

      setup() arguments must *always* be /-separated paths relative to the
      setup.py directory, *never* absolute paths.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for MDAnalysis
Failed to build MDAnalysis
ERROR: Could not build wheels for MDAnalysis, which is required to install pyproject.toml-based projects

Note that I didn't want an editable install pip install -e, which actually works.

p.s.: If my issue is different from @marinegor 's then I can also open a new issue.

[IAlibay]

@orbeckst and @marinegor do your python install environments have a base setuptool / pip installl?

Also @orbeckst could you confirm your macos version?

[marinegor]

@IAlibay how do I check for that? I'm working inside a standard python3 -m venv venvvirtual environment.

[orbeckst]

My env has ~/anaconda3/envs/mda311dask/bin/pip

setuptools                69.5.1             pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
cython                    3.0.10          py311hdd0406b_0    conda-forge

• OS: Darwin deathstar 22.6.0 Darwin Kernel Version 22.6.0: Mon Feb 19 19:48:53 PST 2024; root:xnu-8796.141.3.704.6~1/RELEASE_X86_64 x86_64
• Ventura 13.6.6

[IAlibay]

@IAlibay how do I check for that? I'm working inside a standard python3 -m venv venvvirtual environment.

Not sure 😅, does uv pip list do anything?

[marinegor]

Not sure 😅, does uv pip list do anything?

$ uv pip list
# empty
$ uv pip freeze
# empty

🤷‍♂️🤷‍♂️🤷‍♂️

[yuxuanzhuang]

Processing ~/MDAnalysis/mdanalysis/package

@orbeckst This seems a bit fishy. Is there a tilde in your file path? I think Python does not handle paths with tildes very well. I cannot reproduce the error with my mac (sonoma 14.4). My output for this line is Processing /Users/username/MDAnalysis/mdanalysis/package

[orbeckst]

No, that was me manually sanitizing paths before posting. These are full absolute paths in the real output.

[yuxuanzhuang]

No, that was me manually sanitizing paths before posting. These are full absolute paths in the real output.

That's what I suspected :)

What if you remove the package/MDAnalysis.egg-info and try to install it again?

[orbeckst]

@yuxuanzhuang I deleted the existing MDAnalysis.egg-info and repeated pip install . and I now succeeded

$ pip install .
Processing ~/MDAnalysis/mdanalysis/package
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

...

Building wheels for collected packages: MDAnalysis
  Building wheel for MDAnalysis (pyproject.toml) ... done
  Created wheel for MDAnalysis: filename=MDAnalysis-2.8.0.dev0-cp311-cp311-macosx_10_9_x86_64.whl size=44937915 sha256=753a30559ddcbe1dc77c74a5062bb188e1277c09b876641708aaa35250a4ce7f
  Stored in directory: /private/var/folders/sm/37rm_wm16tq9qsf4n5md98ph0000gp/T/pip-ephem-wheel-cache-16mg3cpr/wheels/e5/2d/a7/19ab06b9a8db11b2f6e588001b5e27bb0dae5c7427ff5a7d03
Successfully built MDAnalysis
Installing collected packages: MDAnalysis
Successfully installed MDAnalysis-2.8.0.dev0

Thank you very much — my problem is solved.