Compatibility with NumPy 2.0

[jorisvandenbossche]

NumPy 2.0 is coming soon, and we should ensure shapely is fully compatible to ensure a smooth transition of our users. Overview issue on numpy's side:

• NumPy 2.0 development status & announcements numpy/numpy#24300

Quick summary of the most important aspects for us:

• NumPy 2.0 as a major release changes the C ABI, so any package that builds against the NumPy C API like shapely will have to be rebuilt with numpy 2.0 to be able to run with numpy 2.0. That also means that our current released packages (built with older versions of numpy) will break (fail to import) with numpy 2.0, and so we should ideally add a numpy<2 pin.
  This happens at the moment when installing shapely from a wheel in an environment with numpy nightly.

• Since numpy 1.25, we don't need oldest-supported-numpy anymore (https://numpy.org/devdocs/release/1.25.0-notes.html#compiling-against-the-numpy-c-api-is-now-backwards-compatible-by-default and https://github.com/scipy/oldest-supported-numpy#deprecation-notice-for-numpy-20), so for newer Python versions (where the minimum supported version of numpy is <=1.25), we can start compiling with the latest numpy.
  This also helps with the point above, because once numpy 2.0 is released, we can therefore built against numpy 2.0 and have a resulting wheel that can run with both numpy 1.x and 2.0. See https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for more details.
• Additionally to C ABI changes, there are also a whole set of changes in the Python API, and so we should test with numpy nightly to ensure we are compatible on that front.

We don't yet test with numpy nightly

A to do list based on my current understanding:

• Start testing with numpy nightly (we currently only use --pre in the dev build, so we only get a dev numpy when they do a beta or RC release). This will ensure that shapely at least builds and runs fine with the upcoming numpy. -> TST: test against numpy nightly #1973
• We should ideally do another release of shapely that adds a numpy<2 pin, so that when numpy 2.0 is out and someone does pip install shapely, you don't get a broken env. Therefore, for the maintenance 2.0.x branch:
  • Add a numpy<2 pin to the requirements in the maintenance branch -> COMPAT: Add a numpy<2 pin to install requirements for 2.0.x #1975
  • Make another 2.0.x release
• For the main development branch, we don't need a runtime pin but we need to update the build dependencies for packaging:
  • Update the build-system requires to use latest numpy instead of oldest-supported-numpy -> BLD: update build-system requires to use latest numpy instead of oldest-supported-numpy + build nightly wheels with numpy nightly #1974
  • Upload nightly wheels when possible (depending on Request access for shapely scientific-python/upload-nightly-action#63)
• Address copy changes in (as)array and __array__
  • COMPAT: fix assert_geometries_equal to work with numpy 2.0 copy changes #2007
  • Update __array__ to accept a copy keyword

[djhoese]

I don't see it mentioned above, but a recent change in numpy 2 seems to be that Cython modules (or anything using the numpy C API) must call np.import_array() at the top of the module. I have CI that installs shapely from github main and it now fails (used to pass with older numpy 2 nightly wheels) because of this upstream change in numpy 2.

Edit: It looks like shapely already runs tests against numpy 2 and is passing:

https://github.com/shapely/shapely/actions/runs/7884170432/job/21512589917?pr=1990

Now I'm very confused at why my CI is failing.

Edit 2: I suppose this means libgeos needs to be compiled with numpy 2 in my CI?

    import shapely.geometry as sgeom
../../../miniconda3/envs/test-environment/lib/python3.12/site-packages/shapely/__init__.py:1: in <module>
    from shapely.lib import GEOSException  # NOQA
E   ImportError: numpy.core.multiarray failed to import

[jorisvandenbossche]

@djhoese do you have a link to the CI in question? (failing build)

[djhoese]

@jorisvandenbossche Here you go: https://github.com/pytroll/pyresample/actions/runs/7906385079/job/21581090150

RuntimeError: module compiled against ABI version 0x1000009 but this version of numpy is 0x2000000

[jorisvandenbossche]

The problem is that pip by default will create an isolated build environment, and at that point it still installs numpy 1.x in that build environment.

So one option is to change the line for shapely at https://github.com/pytroll/pyresample/blob/df92fb921f9116a819c21d75d8e6dba291622e4b/.github/workflows/ci.yaml#L66 to be pip install --no-deps --no-build-isolation git+https://github.com/shapely/shapely

Then that should ensure that shapely is built against the numpy 2.0 nightly you installed just before.

Another option is to figure out how to ensure pip will also install nightly packages from the extra index during the build isolation, but that I don't know exactly what that would look like.

[pllim]

FWIW we found that --pre helped, at least in tox.

https://github.com/astropy/astropy/blob/2c98602609aaf7d41ef9b2b4416d17665d3910d0/tox.ini#L120

[djhoese]

@jorisvandenbossche Thanks. What's sad is I actually knew I needed --no-build-isolation as I've used it in another project for this same exact purpose, but thought I had already done it in pyresample. Tests are done yet, but I'm sure this will fix it. Thanks.

[jorisvandenbossche]

I am planning to do a 2.0.4 release next week to have wheels compatible with numpy 2.0

[jorisvandenbossche]

I am planning to do a 2.0.4 release next week to have wheels compatible with numpy 2.0

This is done in the meantime: https://github.com/shapely/shapely/releases/tag/2.0.4