Compiling from source picks up a nonexistent numpy and fails

[AgilentGCMS]

I am trying to build netCDF4 1.6.5 from source with pip install .. It is failing with a numpy error, namely

Processing /home/sbasu1/Downloads/sources/netCDF4-1.6.5
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [59 lines of output]
      Package hdf5 was not found in the pkg-config search path.
      Perhaps you should add the directory containing `hdf5.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'hdf5', required by 'virtual:world', not found
      reading from setup.cfg...
      using /home/sbasu1/packages/bin/nc-config...
      checking /home/sbasu1/packages/include ...
      HDF5 library version: 1.14.0 headers found in /home/sbasu1/packages/include
      Traceback (most recent call last):
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/__init__.py", line 23, in <module>
          from . import multiarray
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in <module>
          from . import overrides
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/overrides.py", line 6, in <module>
          from numpy.core._multiarray_umath import (
      ImportError: /tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so: failed to map segment from shared object

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/sbasu1/packages/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/sbasu1/packages/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/sbasu1/packages/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 366, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 543, in <module>
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/__init__.py", line 140, in <module>
          from . import core
        File "/tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/__init__.py", line 49, in <module>
          raise ImportError(msg)
      ImportError:

      IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

      Importing the numpy C-extensions failed. This error can happen for
      many reasons, often due to issues with your setup or how NumPy was
      installed.

      We have compiled some common reasons and troubleshooting tips at:

          https://numpy.org/devdocs/user/troubleshooting-importerror.html

      Please note and check the following:

        * The Python version is: Python3.11 from "/home/sbasu1/packages/bin/python3"
        * The NumPy version is: "1.23.2"

      and make sure that they are the versions you expect.
      Please carefully study the documentation linked above for further help.

      Original error was: /tmp/pip-build-env-36b_0j_r/overlay/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so: failed to map segment from shared object

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

What is baffling to me is the numpy version 1.23.2. That is not the numpy version I have installed.

$ python -c 'import numpy; print(numpy.__version__)'
1.26.4

I can in fact import numpy and work with numpy functions. I have scoured my system and that's the only version of numpy I have. So where is this 1.23.2 coming from, that is not letting me install netCDF4?

[jswhit]

don't know why that error messages flags a non-existant numpy version. Looks like the fundamental problem is that the HDF5 library is not found. The docs describe how to tell setuptools to find the C lib dependencies, either via the nc-config utility or by setting environment variables.

[AgilentGCMS]

That's strange. I did export HDF5_DIR to the appropriate variable. I have to say I find this new pip install mechanism (for all packages) very frustrating. The python setup.py route was far more transparent in showing me what errors were occurring where.

[DWesl]

netcdf4-python reports a build dependency on oldest-supported-numpy

netcdf4-python/pyproject.toml

Line 4 in 49dcd0b

"oldest-supported-numpy ; python_version < '3.9'",

which for python3.11 is 1.23.2
https://github.com/scipy/oldest-supported-numpy/blob/23ece5a161d5957f5142991393d2ae689ee45b9d/setup.cfg#L62

Recent pip is "helpful" and will install all build requirements specified in pyproject.toml for a source package into a virtual environment for that build. I think --no-build-isolation will turn that off and build everything in your current python environment.

Also looks like it can't find the file hdf5.pc, as you noticed, and suggests adding directories to PKG_CONFIG_PATH, perhaps ${HDF5_DIR}/lib/pkgconfig or similar?

EDIT: Just noticed 3.11 is not less than 3.9, so pip would install NumPy 2.0.0rc1:

netcdf4-python/pyproject.toml

Line 5 in 49dcd0b

"numpy>=2.0.0rc1 ; python_version >= '3.9'",

except the traceback reports netcdf4-python 1.6.5, not master, so the relevant file is actually

netcdf4-python/pyproject.toml

Line 4 in df470a5

"oldest-supported-numpy",

pip install --no-build-isolation . should tell pip to quit being helpful and just build the package in the current environment.