CI workflow failures with numpy>=2.dev0

[ivilata]

Since Nov 8th, workflow runs of CI are failing in job "ubuntu-latest 3.12 numpy>=2.dev0" on master (the first such failure here). Since previous runs of that job after PR #1068 was merged with support for NumPy 2 nightlies did succeed, it's possible that some API or behaviour of NumPy changed in the meanwhile, causing the breakage.

There are two kinds of errors, one in various tests triggered by errors like "numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'floor_divide' output from dtype('float64') to dtype('uint64') with casting rule 'same_kind'" (which are found in tables/index.py:L629/L625 final_idx32, L584 initial_append, L2005 get_chunkmap, and may be easy to fix with a simple type conversion), the other in test_expression.MixedContainers... (where some array comparisons fail and may be trickier to fix).

Maybe @graingert would be as kind as to have a look at this, since his work on supporting NumPy 2 is recent enough?

Thanks a lot! 🙂

[avalentino]

I have a working branche for it: https://github.com/avalentino/PyTables/tree/feature/fix-numpy2-20231112

Unfortunately there are still some test failures.

Moreover I see here:

• support new numpy complex types pydata/numexpr#458
• ENH: Port scipy to use new numpy complex types scipy/scipy#18975
• ENH: Future of npymath library numpy/numpy#20880
• ENH: Replace npy complex structs with native complex types numpy/numpy#24085

that there could be different solutions to be considered for fixing the "complex" related issues.

[ivilata]

Thanks @avalentino for taking care of this! Since the issue is quite well isolated in that CI job while the rest of workflows and jobs do succeed (including the creation of wheels), and since the issue with b2nd slicing mentioned in #1078 is already fixed in master, I think I'll go ahead and create a PR for a new release v3.9.2. 🙂

[avalentino]

OK, for me.
Please go on with your PR

Just I would like to point out that I did some work on the numpy2 issue but at the moment I'm not after it due to lack of time. So if anyone want to take over he is very welcome

[lagru]

Hey, @jarrodmillman pointed out this issue to me as I've recently upgraded scikit-image to work with both NumPy 1.x and 2.0 (see scikit-image/scikit-image#7288). Your are welcome to ping me if you have concrete problems related to that, I'll have a look and see if I can add anything useful. 🤞

[avalentino]

Thanks @lagru , very appreciated

[rgommers]

NumPy 2.0.0rc1 is released, which means you can do a release that is built against it.

A quick note on dependency metadata: oldest-supported-numpy should be replaced by numpy>=2.0.0rc1 in the [build-system] table in pyproject.toml. That'll yield wheels that are compatible with 1.x and 2.x.

[maxnoe]

One issue is the change of complex types:

src/utils.c: In function ‘create_ieee_complex64’:
src/utils.c:768:30: error: request for member ‘real’ in something not a structure or union
  768 |   H5Tinsert(complex_id, "r", HOFFSET(npy_complex64, real), float_id);
      |                              ^~~~~~~
src/utils.c:769:30: error: request for member ‘imag’ in something not a structure or union
  769 |   H5Tinsert(complex_id, "i", HOFFSET(npy_complex64, imag), float_id);
      |                              ^~~~~~~
src/utils.c: In function ‘create_ieee_complex128’:
src/utils.c:793:30: error: request for member ‘real’ in something not a structure or union
  793 |   H5Tinsert(complex_id, "r", HOFFSET(npy_complex128, real), float_id);
      |                              ^~~~~~~
src/utils.c:794:30: error: request for member ‘imag’ in something not a structure or union
  794 |   H5Tinsert(complex_id, "i", HOFFSET(npy_complex128, imag), float_id);
      |                              ^~~~~~~
src/utils.c: In function ‘create_ieee_complex192’:
src/utils.c:825:30: error: request for member ‘real’ in something not a structure or union
  825 |   H5Tinsert(complex_id, "r", HOFFSET(npy_complex192, real), float_id);
      |                              ^~~~~~~
src/utils.c:826:30: error: request for member ‘imag’ in something not a structure or union
  826 |   H5Tinsert(complex_id, "i", HOFFSET(npy_complex192, imag), float_id);
      |                              ^~~~~~~
src/utils.c: In function ‘create_ieee_complex256’:
src/utils.c:857:30: error: request for member ‘real’ in something not a structure or union
  857 |   H5Tinsert(complex_id, "r", HOFFSET(npy_complex256, real), float_id);
      |                              ^~~~~~~
src/utils.c:858:30: error: request for member ‘imag’ in something not a structure or union
  858 |   H5Tinsert(complex_id, "i", HOFFSET(npy_complex256, imag), float_id);
      |                              ^~~~~~~

Any idea how to address this?

See: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#complex-types-underlying-type-changes

[avalentino]

I have a work-in-progress branch (https://github.com/avalentino/PyTables/tree/feature/fix-numpy2-20231112).
Unfortunately I do not have time to finalise it at the moment.