Random CI failure on MacOS: RuntimeWarning: invalid value encountered in reciprocal

[crusaderky]

This has been failing multiple times this afternoon alone. It seems that the failures are concentrated on macos 3.8 but I'm not 100% sure about it.
https://github.com/dask/dask/pull/7109/checks?check_run_id=1856104314
https://github.com/dask/dask/pull/7177/checks?check_run_id=1856169384

_________________ test_norm_any_slice[True--1-shape3-chunks3] __________________
[gw2] darwin -- Python 3.8.6 /Users/runner/miniconda3/envs/test-environment/bin/python

shape = (4, 5, 2, 3), chunks = (2, 2, 2, 2), norm = -1, keepdims = True

    @pytest.mark.slow
    @pytest.mark.parametrize(
        "shape, chunks",
        [
            [(5,), (2,)],
            [(5, 3), (2, 2)],
            [(4, 5, 3), (2, 2, 2)],
            [(4, 5, 2, 3), (2, 2, 2, 2)],
            [(2, 5, 2, 4, 3), (2, 2, 2, 2, 2)],
        ],
    )
    @pytest.mark.parametrize("norm", [None, 1, -1, np.inf, -np.inf])
    @pytest.mark.parametrize("keepdims", [False, True])
    def test_norm_any_slice(shape, chunks, norm, keepdims):
        a = np.random.random(shape)
        d = da.from_array(a, chunks=chunks)
    
        for firstaxis in range(len(shape)):
            for secondaxis in range(len(shape)):
                if firstaxis != secondaxis:
                    axis = (firstaxis, secondaxis)
                else:
                    axis = firstaxis
                a_r = np.linalg.norm(a, ord=norm, axis=axis, keepdims=keepdims)
                d_r = da.linalg.norm(d, ord=norm, axis=axis, keepdims=keepdims)
>               assert_eq(a_r, d_r)

dask/array/tests/test_linalg.py:1014: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dask/array/utils.py:264: in assert_eq
    b, bdt, b_meta, b_computed = _get_dt_meta_computed(
dask/array/utils.py:239: in _get_dt_meta_computed
    x = x.compute(scheduler="sync")
dask/base.py:282: in compute
    (result,) = compute(self, traverse=False, **kwargs)
dask/base.py:564: in compute
    results = schedule(dsk, keys, **kwargs)
dask/local.py:528: in get_sync
    return get_async(apply_sync, 1, dsk, keys, **kwargs)
dask/local.py:495: in get_async
    fire_task()
dask/local.py:457: in fire_task
    apply_async(
dask/local.py:517: in apply_sync
    res = func(*args, **kwds)
dask/local.py:227: in execute_task
    result = pack_exception(e, dumps)
dask/local.py:222: in execute_task
    result = _execute_task(task, data)
dask/core.py:121: in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:121: in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:121: in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:121: in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:121: in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
dask/optimization.py:963: in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
dask/core.py:151: in get
    result = _execute_task(task, cache)
dask/core.py:121: in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:121: in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
dask/core.py:115: in _execute_task
    return [_execute_task(a, cache) for a in arg]
dask/core.py:115: in <listcomp>
    return [_execute_task(a, cache) for a in arg]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

arg = (<built-in function pow>, (<built-in function abs>, '_1'), '_0')
cache = {'_0': -1, '_1': array([[[[0.98560541],
         [0.59956337]],

        [[0.66212237],
         [0.92693976]]],


       [[[0.97698657],
         [0.30799226]],

        [[0.10911509],
         [0.89820523]]]])}
dsk = None

    def _execute_task(arg, cache, dsk=None):
        """Do the actual work of collecting data and executing a function
    
        Examples
        --------
    
        >>> cache = {'x': 1, 'y': 2}
    
        Compute tasks against a cache
        >>> _execute_task((add, 'x', 1), cache)  # Compute task in naive manner
        2
        >>> _execute_task((add, (inc, 'x'), 1), cache)  # Support nested computation
        3
    
        Also grab data from cache
        >>> _execute_task('x', cache)
        1
    
        Support nested lists
        >>> list(_execute_task(['x', 'y'], cache))
        [1, 2]
    
        >>> list(map(list, _execute_task([['x', 'y'], ['y', 'x']], cache)))
        [[1, 2], [2, 1]]
    
        >>> _execute_task('foo', cache)  # Passes through on non-keys
        'foo'
        """
        if isinstance(arg, list):
            return [_execute_task(a, cache) for a in arg]
        elif istask(arg):
            func, args = arg[0], arg[1:]
            # Note: Don't assign the subtask results to a variable. numpy detects
            # temporaries by their reference count and can execute certain
            # operations in-place.
>           return func(*(_execute_task(a, cache) for a in args))
E           RuntimeWarning: invalid value encountered in reciprocal

[crusaderky]

This issue is caused by the specific combination of MacOSX AND numpy 1.20 AND conda-forge.
Upstream ticket: conda-forge/numpy-feedstock#229

[jrbourbeau]

For reference, in #7211 we updated CI to temporarily install NumPy from PyPI for Python 3.8 build on Mac. Let's continue to engage upstream and revert #7211 once conda-forge/numpy-feedstock#229 has been resolved

[jakirkham]

Just to add here this traces back to NumPy issue ( numpy/numpy#18555 ) as mentioned in comment ( conda-forge/numpy-feedstock#229 (comment) ). This was fixed in PR ( numpy/numpy#19926 ) included in NumPy 1.22.0, which was recently released in conda-forge ( conda-forge/numpy-feedstock#251 )

[crusaderky]

@jrbourbeau could you retest on your macintel please?

conda create -n test -c conda-forge python=3.8 numpy=1.22
conda activate test
python -c "import numpy; numpy.ones(120) ** -1"

[jrbourbeau]

Yep, I can confirm python -c "import numpy; numpy.ones(120) ** -1" no longer raises a RuntimeWarning with the setup in #7189 (comment)