BUG: pytest fails with ModuleNotFoundError: No module named 'utils'

[tuhinsharma121]

Describe the issue:

I have followed the official build commands for editable install. But pytest fails with ModuleNotFoundError: No module named 'utils'

Reproduce the code example:

conda env create --file environment.yml
conda activate numpy-dev
git fetch upstream --tags
pip install --editable . --no-build-isolation  --config-settings editable-verbose=true
pytest numpy

Error message:

=============================================================================================== test session starts ===============================================================================================
platform darwin -- Python 3.11.9, pytest-7.4.0, pluggy-1.5.0
rootdir: /Users/tuhinsharma/Documents/Git/numpy
configfile: pytest.ini
plugins: hypothesis-6.81.1, cov-4.1.0, xdist-3.5.0
collected 51576 items / 1 error / 2 skipped                                                                                                                                                                       

===================================================================================================== ERRORS ======================================================================================================
____________________________________________________________ ERROR collecting numpy/_core/src/common/pythoncapi-compat/tests/test_pythoncapi_compat.py ____________________________________________________________
ImportError while importing test module '/Users/tuhinsharma/Documents/Git/numpy/numpy/_core/src/common/pythoncapi-compat/tests/test_pythoncapi_compat.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/numpy-dev/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
numpy/_core/src/common/pythoncapi-compat/tests/test_pythoncapi_compat.py:24: in <module>
    from utils import run_command, command_stdout
E   ModuleNotFoundError: No module named 'utils'
============================================================================================= short test summary info =============================================================================================
ERROR numpy/_core/src/common/pythoncapi-compat/tests/test_pythoncapi_compat.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================================================== 2 skipped, 1 error in 8.73s ===========================================================================================

Python and NumPy Versions:

Python 3.11.9
-e git+ssh://git@github.com/tuhinsharma121/numpy.git@552fdd4a476c6cc99a9f985493664f7279bbff65#egg=numpy

Runtime Environment:

import numpy; print(numpy.show_runtime())

meson-python: building numpy: /opt/anaconda3/envs/numpy-dev/bin/ninja
[1/1] Generating numpy/generate-version with a custom command
Saving version to numpy/version.py
[{'numpy_version': '2.1.0.dev0+git20240421.552fdd4',
'python': '3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:45:13) '
'[Clang 16.0.6 ]',
'uname': uname_result(system='Darwin', node='tuhsharm-mac', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3', 'SSSE3'],
'found': ['SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}}]
None

Context for the issue:

I want to contribute to the project. Not getting my local test work is a blocker for it.

[mattip]

Those tests are in a submodule pulled in recently, and the tests cannot work without some refactoring. I wonder why our CI does not collect these and fail. @vstinner thoughts about how to work around this? Should we delete the tests?

[ngoldbaum]

I wonder why pytest doesn’t see

numpy/pytest.ini

Line 3 in 3d33d5f

norecursedirs = doc tools numpy/linalg/lapack_lite numpy/_core/code_generators numpy/_core/src/common/pythoncapi-compat

[vstinner]

@vstinner thoughts about how to work around this? Should we delete the tests?

I'm not used to pytest configuration. Obviously, yes, removing tests solve the issue.

[mattip]

I tried the commands at the top of the issue and the problem does not reproduce for me (on a macbook with arm64 processsor).

@tuhinsharma121 could you use some of the pytest options to try to figure out why the pytest.ini is ignored? Do you have anything special in your environment? Maybe try copying the norecursedirs value in a --override-ini command line option to pytest.

[tuhinsharma121]

I pulled the latest code.

pytest numpy

works fine from the terminal now.

[tuhinsharma121]

I see another problem now. It runs fine from the terminal. But when I configure pytest from pycharm it fails with following error

/opt/anaconda3/envs/numpy-dev/bin/python /Users/tuhinsharma/Applications/PyCharm Professional.app/Contents/plugins/python/helpers/pycharm/_jb_pytest_runner.py -- numpy 
Testing started at 8:40 am ...

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.0.dev0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<frozen site>", line 626, in <module>
  File "<frozen site>", line 619, in main
  File "<frozen site>", line 558, in execsitecustomize
  File "/Users/tuhinsharma/Applications/PyCharm Professional.app/Contents/plugins/python/helpers/pycharm_matplotlib_backend/sitecustomize.py", line 43, in <module>
    import matplotlib
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/__init__.py", line 161, in <module>
    from . import _api, _version, cbook, _docstring, rcsetup
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/rcsetup.py", line 27, in <module>
    from matplotlib.colors import Colormap, is_color_like
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/colors.py", line 57, in <module>
    from matplotlib import _api, _cm, cbook, scale
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/scale.py", line 22, in <module>
    from matplotlib.ticker import (
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/ticker.py", line 143, in <module>
    from matplotlib import transforms as mtransforms
  File "/opt/anaconda3/envs/numpy-dev/lib/python3.11/site-packages/matplotlib/transforms.py", line 49, in <module>
    from matplotlib._path import (
AttributeError: _ARRAY_API not found
Launching pytest with arguments numpy --no-header --no-summary -q in /Users/tuhinsharma/Documents/Git/numpy

ImportError while loading conftest '/Users/tuhinsharma/Documents/Git/numpy/numpy/conftest.py'.
numpy/__init__.py:525: in <module>
    _core.multiarray._multiarray_umath._reload_guard()
E   UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.

Process finished with exit code 4

Empty suite

Empty suite

It looks like pytest runner in pycharm has a dependency on matplotlib. And matplotlib is built with numpy 1.0 instead 2.0. Hence giving the error. I guess I have to wait for matplotlib to build with numpy 2.0 to use pycharm pytest runner which may take time.

[rgommers]

Please upgrade to Matplotlib 3.8.4, that is built against NumPy 2.0.

The original issue here is resolved, so I'll close this. Thanks all.