BUG: numpy.isin does not function correctly with two arrays with different integer type

[TonyXiang8787]

Describe the issue:

The function numpy.isin sometimes returns the wrong answer, if the integer type of the two arrays (to be compared) are not the same.

The example below shows a int8 array with one zero and a int64 array with two values. It should return one True. However, it returns False.

Reproduce the code example:

>>> import numpy as np
>>> np.isin(np.zeros(1, dtype=np.int8), np.array([-128, 0], dtype=np.int64), kind='table')
array([False])

Error message:

No response

Runtime information:

Numpy version

>>> print(numpy.__version__)
1.24.0

Sys version

>>> print(sys.version)
3.11.1 (main, Dec  7 2022, 01:11:34) [GCC 11.3.0]

Numpy runtime

[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': 'PATH_DELETED_DUE_TO_PRIVACY/.venv/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 20,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

Context for the issue:

The problem seems to only occur in version 1.24.

[nkit-chahal]

NumPy version 1.20.3
it's working fine

[TonyXiang8787]

@nkit-chahal It's a new problem in version 1.24.

[seberg]

@MilesCranmer do you have time to have a quick look. This looks like it must be related to gh-12065 misfiring.
Would be nice to fix for 1.24.1, but considering that needs to go out soon not sure that will happen.

[MilesCranmer]

Sure, I have time to look at this today.

[TonyXiang8787]

@MilesCranmer After some research I think the problem might be in this line:

numpy/numpy/lib/arraysetops.py

Line 683 in 2b9851b

outgoing_array[basic_mask] = isin_helper_ar[ar1[basic_mask] -

If ar1 has a smaller data type than ar2, it may overflow here.

I cannot think about an easy solution here. We can convert ar1 to the same type of ar2, but it can increase memory usage dramatically (int8 to int64 is 8 times bigger). We can check the overflow for ar1 and if overflow can happen we switch back to sort method, but this sacrifices the performance.

[MilesCranmer]

Quick question for @seberg: is this behaviour actually incorrect? In other words, do we want int8(0) == int64(0) in the context of isin, or to have it be !=? I could see either thing being technically correct, I suppose it depends on NumPy conventions. (I guess I could see isin having different behaviour than == if someone has a mixed type array)

[TonyXiang8787]

Quick question for @seberg: is this behaviour actually incorrect? In other words, do we want int8(0) == int64(0) in the context of isin, or to have it be !=? I could see either thing being technically correct, I suppose it depends on NumPy conventions. (I guess I could see isin having different behaviour than == if someone has a mixed type array)

The behavior before version 1.24 is int8(0) == int64(0) in the context of isin.

[MilesCranmer]

@TonyXiang8787 I think the solution should be implemented by correcting the overflow detection here:

numpy/numpy/lib/arraysetops.py

Line 652 in 2b9851b

range_safe_from_overflow = ar2_range < np.iinfo(ar2.dtype).max

To include information about the type of ar1.

[MilesCranmer]

Yep this looks like the solution:

-  range_safe_from_overflow = ar2_range < np.iinfo(ar2.dtype).max 
+  range_safe_from_overflow = ar2_range < np.iinfo(ar2.dtype).max and (ar1_max - ar2_min) < np.iinfo(ar1.dtype).max and (ar1_min - ar2_max) > np.iinfo(ar1.dtype).min

I can make a PR when I get into the office, double check this fixes the bug, and also include some new unit tests.

[MilesCranmer]

PR created on #22878. Thanks for raising this issue @TonyXiang8787.