Numpy 2 compatibility

[neutrinoceros]

I just saw that pyart 1.18.1 still uses oldest-supported-numpy at build time, which indicates that it won't be compatible with the upcoming Numpy 2.0 release. Here's a bit of context:

• Numpy 2.0 is currently at the release candidate stage and is ABI unstable
• this tracking issue contains guidance to update packages that require it at build time NumPy 2.0 development status & announcements numpy/numpy#24300 (comment)

I know that pyart has a lot of big dependencies so it might still take a while before it can be easily tested against numpy 2, however it should already be possible to build future wheels against numpy 2.0 to allow anyone downstream to test it.
The patch would be as simple as

diff --git a/pyproject.toml b/pyproject.toml
index 2744eb91..9c0a8b5b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ requires = [
   "setuptools_scm>=6.2",
   "wheel",
   "cython",
-  "oldest-supported-numpy"
+  "numpy>=2.0.0rc1",
 ]

 [tool.black]

Coordination at the ecosystem level is being tracked at numpy/numpy#26191 (comment), so I recommend keeping an eye out to see when your direct dependencies are compatible and start testing against it, preferably before numpy 2.0.0 final is out.

[zssherman]

Hey @neutrinoceros, thanks for the heads up! @mgrover1 and I will keep an eye out and will get that tested and switched over as soon as possible. Thanks again!

[zssherman]

@neutrinoceros More of a heads up, we were able to get the numpy wheels builds going etc, but we are still trying to figure out the cause of the cython errors in the mac os feedstock builds.

[neutrinoceros]

thanks for letting me know !

[mgrover1]

Reopening - unfortunately we cannot support this yet @neutrinoceros as we are running into conda-forge distribution errors. Until this is tested/sorted out, we are rolling back to oldest-supported-numpy

[neutrinoceros]

Not sure I get what the problem is. My understanding is that conda-forge has its own, unrelated mechanism for dealing with numpy ABI compatibility and numpy 2.0 isn't a concern yet from the standpoint of feedstock maintainers.
xref conda-forge/conda-forge.github.io#1997

[mgrover1]

We are seeing numerous cython related issues
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=917141&view=logs&j=9c5ef928-2cd6-52e5-dbe6-9d173a7d951b&t=20c71c51-4b27-578b-485d-06ade2de1d00&l=1608

[mgrover1]

the only thing that has changed as been moving from numpy 1 to 2

[neutrinoceros]

As far as I understand, conda-forge build bots do not even read build system metadata from pyproject.toml, so I doubt this is where the problem is. However, it should be okay to apply any patch needed in that area directly in your conda-forge recipe, since building against numpy 2.0.0rc1 really just matters for PyPI wheels

[mgrover1]

Ohhh that is a good point - I will apply the patch and see if that resolves the issue.

[mgrover1]

There is an issue with numpy 2.0 and cython.. described here
cython/cython#6100 (comment)

[neutrinoceros]

I see that pip seems to be used in the conda-forge build process, which seems unusual (to me ?), so that might be why, but patching pyproject.toml on conda-forge should be enough to resolve the matter without loosing numpy 2 compat in PyPI wheels.

[mgrover1]

Do you have an example of applying patches to conda-forge recipes?

[dopplershift]

@neutrinoceros It's not weird at all that pip is involved on conda-forge; lots of projects have $PYTHON -m pip install --no-deps --no-build-isolation -vv . as their canonical build step.

@mgrover1 Here's an example where the recipe for netcdf patches the source. Essentially, create a patch file (could be as easy as git diff > my.patch) and list it under sources; conda-build will take care of applying it.

[mgrover1]

Great - thanks @dopplershift !!