Ufunc method incompatibility with Numpy 1.21

[mhostetter]

Using ufunc methods of Numba-generated ufuncs doesn't work with Numpy 1.21. In Numpy 1.21 they added argument checking to ufuncs. It seems to have implications here.

Numpy complains that exactly two arguments are required, but two arguments are passed in. So I'm unsure what the actual error is.

Numpy version: 1.21.0
Numba version: 0.53.1

Minimal example

# File bug.py
import numpy as np
import numba

print("numpy version:", np.__version__)
print("numba version", numba.__version__)

def my_xor(x, y):
    return x ^ y

xor_ufunc = numba.vectorize("int64(int64, int64)", nopython=True)(my_xor)

x = np.array([1, 7], dtype=np.int64)
y = np.array([3, 4], dtype=np.int64)
z = xor_ufunc(x, y)
print("x:", x)
print("y:", y)
print("my_xor(x, y):", z)

z2 = xor_ufunc.outer(x, y)
print("my_xor.outer(x, y):\n", z2)

Output with Numpy 1.20.3

In [1]: %run bug.py                                                                                                                          
numpy version: 1.20.3
numba version 0.53.1
x: [1 7]
y: [3 4]
my_xor(x, y): [2 3]
my_xor.outer(x, y):
 [[2 5]
 [4 3]]

Output with Numpy 1.21.0

In [1]: %run bug.py                                                                                                                          
numpy version: 1.21.0
numba version 0.53.1
x: [1 7]
y: [3 4]
my_xor(x, y): [2 3]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/repos/galois/bug.py in <module>
     18 print("my_xor(x, y):", z)
     19 
---> 20 z2 = xor_ufunc.outer(x, y)
     21 print("my_xor.outer(x, y):\n", z2)

TypeError: exactly two arguments expected

[gmarkall]

Thanks for the report! NumPy 1.21 is not yet supported, but this will be looked at as part of supporting it.

[gmarkall]

#7176 created to track items needed for NumPy 1.21 support.

[github-actions]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[gmarkall]

I've removed needtriage so the stale bot doesn't turn up on this issue again.

[seberg]

I completely randomly stumbled on this issue. It seems fairly likely to me that this is a NumPy issue! Please make sure to bug me/us about this type of thing!
Do you think the reduce crash could be a similar problem?

[seberg]

OK, nevermind. I thought the problem here might be that NumPy does not take into account the possible existance of PY_VECTORCALL_ARGUMENTS_OFFSET (and uses the NARGS macro). This may still be a bug in NumPy, but not what is causing this.

Trying it out, NumPy actually reports 0 arguments being passed. Which means the problem is quite different: It looks like Numba is calling the NumPy method slot directly in C, but simply assumes the method is a normal Python method and not a vectorcall/fastcall one? I guess as a micro-optimization to avoid using PyObject_Call?

The reduction crash seems like exactly the same problem.

EDIT: Or I suppose more likely: Numba copies the function pointer into its own object or so? But fails to also copy the METH_FASTCALL flag?

[seberg]

Yeah, the code below will not work because it does not know about vectorcall. That is a bit unfortunate, I hope numba can easily catch up and use the vectorcall protocol on newer NumPy versions.

numba/numba/np/ufunc/_internal.c

Lines 289 to 351 in 3927115

	init_ufunc_dispatch(void)
	{
	int result = 0;
	PyMethodDef * crnt = PyUFunc_Type.tp_methods;
	const char * crnt_name = NULL;
	for (; crnt->ml_name != NULL; crnt++) {
	crnt_name = crnt->ml_name;
	switch(crnt_name[0]) {
	case 'a':
	if (strncmp(crnt_name, "accumulate", 11) == 0) {
	ufunc_dispatch.ufunc_accumulate =
	(PyCFunctionWithKeywords)crnt->ml_meth;
	#if NPY_API_VERSION >= 0x00000008
	} else if (strncmp(crnt_name, "at", 3) == 0) {
	ufunc_dispatch.ufunc_at = crnt->ml_meth;
	#endif
	} else {
	result = -1;
	}
	break;
	case 'o':
	if (strncmp(crnt_name, "outer", 6) == 0) {
	ufunc_dispatch.ufunc_outer =
	(PyCFunctionWithKeywords)crnt->ml_meth;
	} else {
	result = -1;
	}
	break;
	case 'r':
	if (strncmp(crnt_name, "reduce", 7) == 0) {
	ufunc_dispatch.ufunc_reduce =
	(PyCFunctionWithKeywords)crnt->ml_meth;
	} else if (strncmp(crnt_name, "reduceat", 9) == 0) {
	ufunc_dispatch.ufunc_reduceat =
	(PyCFunctionWithKeywords)crnt->ml_meth;
	} else {
	result = -1;
	}
	break;
	default:
	result = -1; /* Unknown method */
	}
	if (result < 0) break;
	}
	if (result == 0) {
	/* Sanity check. */
	result = ((ufunc_dispatch.ufunc_reduce != NULL)
	&& (ufunc_dispatch.ufunc_accumulate != NULL)
	&& (ufunc_dispatch.ufunc_reduceat != NULL)
	&& (ufunc_dispatch.ufunc_outer != NULL)
	#if NPY_API_VERSION >= 0x00000008
	&& (ufunc_dispatch.ufunc_at != NULL)
	#endif
	);
	}
	return result;
	}
	static PyObject *
	dufunc_reduce(PyDUFuncObject * self, PyObject * args, PyObject *kws)
	{
	return ufunc_dispatch.ufunc_reduce((PyObject*)self->ufunc, args, kws);
	}

[rekado]

I'm just a curious bystander, but I'd like to note that numpy 1.21.3 will become the default numpy in GNU Guix in a few days. We'll keep around numpy 1.20 specifically for numba, but it will complicate the installation of numba together with other Python packages using numpy 1.21 by default.

I'd be happy to upgrade numba in Guix as soon as compatibility with numpy 1.21 is achieved. (Subscribing to this issue to be get notification when this happens.)

Good luck!

[stuartarchibald]

@rekado Numba 0.55 release candidates will be out within the next week or so (hopefully!), these will have NumPy 1.21.x series support. A post will be made to https://numba.discourse.group/c/announcements/ when RC builds are shipped.

[stuartarchibald]

Fixed by the now merged #7483, which includes the C code changes from #7449. Closing.