MAINT: 1.9.3 backports

doc/release/1.9.3-notes.rst

@@ -10,10 +10,113 @@ compared to 1.9.2.
 Authors
 =======
 
+* Jelle Aalbers (1)
+* Peter Bell (1)
+* Jake Bowhay (3)
+* Matthew Brett (3)
+* Evgeni Burovski (5)
+* drpeteb (1) +
+* Sebastian Ehlert (1) +
+* GavinZhang (1) +
+* Ralf Gommers (2)
+* Matt Haberland (15)
+* Lakshaya Inani (1) +
+* Joseph T. Iosue (1)
+* Nathan Jacobi (1) +
+* jmkuebler (1) +
+* Nikita Karetnikov (1) +
+* Lechnio (1) +
+* Nicholas McKibben (1)
+* Andrew Nelson (1)
+* o-alexandre-felipe (1) +
+* Tirth Patel (1)
+* Tyler Reddy (51)
+* Martin Reinecke (1)
+* Marie Roald (1) +
+* Pamphile Roy (2)
+* Eli Schwartz (1)
+* serge-sans-paille (1)
+* ehsan shirvanian (1) +
+* Mamoru TASAKA (1) +
+* Samuel Wallan (1)
+* Warren Weckesser (7)
+* Gavin Zhang (1) +
+
+A total of 31 people contributed to this release.
+People with a "+" by their names contributed a patch for the first time.
+This list of names is automatically generated, and may not be fully complete.
 
 Issues closed for 1.9.3
 -----------------------
 
+* `#3691 <https://github.com/scipy/scipy/issues/3691>`__: scipy.interpolate.UnivariateSpline segfault
+* `#5286 <https://github.com/scipy/scipy/issues/5286>`__: BUG: multivariate_normal returns a pdf for values outside its...
+* `#6551 <https://github.com/scipy/scipy/issues/6551>`__: BUG: stats: inconsistency in docs and behavior of gmean and hmean
+* `#9245 <https://github.com/scipy/scipy/issues/9245>`__: running scipy.interpolate.tests.test_fitpack::test_bisplev_integer_overflow...
+* `#12471 <https://github.com/scipy/scipy/issues/12471>`__: test_bisplev_integer_overflow: Segmentation fault (core dumped)
+* `#13321 <https://github.com/scipy/scipy/issues/13321>`__: Bug: setting iprint=0 hides all output from fmin_l_bfgs_b, but...
+* `#13730 <https://github.com/scipy/scipy/issues/13730>`__: \`scipy.stats.mood\` does not correct for ties
+* `#14019 <https://github.com/scipy/scipy/issues/14019>`__: ks_2samp throws \`RuntimeWarning: overflow encountered in double_scalars\`
+* `#14589 <https://github.com/scipy/scipy/issues/14589>`__: \`shgo\` error since scipy 1.8.0.dev0+1529.803e52d
+* `#14591 <https://github.com/scipy/scipy/issues/14591>`__: Input data validation for RectSphereBivariateSpline
+* `#15101 <https://github.com/scipy/scipy/issues/15101>`__: BUG: binom.pmf - RuntimeWarning: divide by zero
+* `#15342 <https://github.com/scipy/scipy/issues/15342>`__: BUG: scipy.optimize.minimize: Powell's method function evaluated...
+* `#15964 <https://github.com/scipy/scipy/issues/15964>`__: BUG: lombscargle fails if argument is a view
+* `#16211 <https://github.com/scipy/scipy/issues/16211>`__: BUG: Possible bug when using winsorize on pandas data instead...
+* `#16459 <https://github.com/scipy/scipy/issues/16459>`__: BUG: stats.ttest_ind returns wrong p-values with permutations
+* `#16500 <https://github.com/scipy/scipy/issues/16500>`__: odr.Model default meta value fails with __getattr__
+* `#16519 <https://github.com/scipy/scipy/issues/16519>`__: BUG: Error in error message for incorrect sample dimension in...
+* `#16527 <https://github.com/scipy/scipy/issues/16527>`__: BUG: dimension of isuppz in syevr is mistranslated
+* `#16600 <https://github.com/scipy/scipy/issues/16600>`__: BUG: \`KDTree\`'s optional argument \`eps\` seems to have no...
+* `#16656 <https://github.com/scipy/scipy/issues/16656>`__: dtype not preserved with operations on sparse arrays
+* `#16751 <https://github.com/scipy/scipy/issues/16751>`__: BUG: \`stats.fit\` on \`boltzmann\` expects \`bound\` for \`lambda\`,...
+* `#17012 <https://github.com/scipy/scipy/issues/17012>`__: BUG: Small oversight in sparse.linalg.lsmr?
+* `#17020 <https://github.com/scipy/scipy/issues/17020>`__: BUG: Build failure due to problems with shebang line in cythoner.py
+* `#17088 <https://github.com/scipy/scipy/issues/17088>`__: BUG: stats.rayleigh.fit: returns \`loc\` that is inconsistent...
+* `#17104 <https://github.com/scipy/scipy/issues/17104>`__: BUG? Incorrect branch in \`LAMV\` / \`_specfunc.lamv\`
+* `#17196 <https://github.com/scipy/scipy/issues/17196>`__: DOC: keepdims in stats.mode is incorrectly documented
+
 
 Pull requests for 1.9.3
 -----------------------
+
+* `#5288 <https://github.com/scipy/scipy/pull/5288>`__: BUG: multivariate_normal returns a pdf for values outside its...
+* `#13322 <https://github.com/scipy/scipy/pull/13322>`__: Bug: setting iprint=0 hides all output from fmin_l_bfgs_b, but...
+* `#13349 <https://github.com/scipy/scipy/pull/13349>`__: BUG: stats: Reformulate loggamma._rvs to handle c << 1.
+* `#13411 <https://github.com/scipy/scipy/pull/13411>`__: BUG: fix out-of-bound evaluations in optimize.minimize, powell...
+* `#15363 <https://github.com/scipy/scipy/pull/15363>`__: BUG: fix powell evaluated outside limits
+* `#15381 <https://github.com/scipy/scipy/pull/15381>`__: BUG: fix stats.rv_histogram for non-uniform bins
+* `#16212 <https://github.com/scipy/scipy/pull/16212>`__: stats.mood: correct for when ties are present
+* `#16288 <https://github.com/scipy/scipy/pull/16288>`__: BUG: fix a crash in \`fpknot\`
+* `#16318 <https://github.com/scipy/scipy/pull/16318>`__: MAINT: stats: fix _contains_nan on Pandas Series
+* `#16460 <https://github.com/scipy/scipy/pull/16460>`__: Fix ttest permutations
+* `#16506 <https://github.com/scipy/scipy/pull/16506>`__: MAINT: fix SHGO extra arguments
+* `#16521 <https://github.com/scipy/scipy/pull/16521>`__: BUG: Fix error in error message for incorrect sample dimension...
+* `#16525 <https://github.com/scipy/scipy/pull/16525>`__: MAINT: stats.ks_2samp: always emit warning when exact method...
+* `#16528 <https://github.com/scipy/scipy/pull/16528>`__: BUG: fix syevr series segfault by explicitly specifying operator...
+* `#16562 <https://github.com/scipy/scipy/pull/16562>`__: BUG: optimize: Fix differential_evolution error message.
+* `#16573 <https://github.com/scipy/scipy/pull/16573>`__: FIX: \`odr.Model\` error with default \`meta\` value
+* `#16588 <https://github.com/scipy/scipy/pull/16588>`__: FIX: stats: ignore divide-by-zero warnings from Boost binom impl
+* `#16590 <https://github.com/scipy/scipy/pull/16590>`__: MAINT: stats.vonmises: wrap rvs to -pi, pi interval
+* `#16630 <https://github.com/scipy/scipy/pull/16630>`__: BUG: eps param no effect fixed
+* `#16645 <https://github.com/scipy/scipy/pull/16645>`__: MAINT: Ensure Pythran input for lombscargle are contiguous
+* `#16649 <https://github.com/scipy/scipy/pull/16649>`__: Detect integer overflow in bivariate splines in fitpackmodule.c,...
+* `#16657 <https://github.com/scipy/scipy/pull/16657>`__: BUG: sparse: Fix indexing sparse matrix with empty index arguments.
+* `#16669 <https://github.com/scipy/scipy/pull/16669>`__: FIX: spurious divide error with \`gmean\`
+* `#16701 <https://github.com/scipy/scipy/pull/16701>`__: BUG: fix mutable data types as default arguments in \`ord.{Data,RealData}\`
+* `#16752 <https://github.com/scipy/scipy/pull/16752>`__: MAINT: stats.boltzmann: correct _shape_info typo
+* `#16780 <https://github.com/scipy/scipy/pull/16780>`__: BUG: interpolate: sanity check x and y in make_interp_spline(x,...
+* `#16836 <https://github.com/scipy/scipy/pull/16836>`__: MAINT: avoid \`func_data\`, it conflicts with system header on...
+* `#16872 <https://github.com/scipy/scipy/pull/16872>`__: BUG: interpolate: work array sizes for RectSphereBivariateSpline
+* `#16965 <https://github.com/scipy/scipy/pull/16965>`__: BUG: linalg: Fix the XSLOW test test_sgesdd_lwork_bug_workaround()
+* `#17043 <https://github.com/scipy/scipy/pull/17043>`__: MAINT: fix small LSMR problem
+* `#17090 <https://github.com/scipy/scipy/pull/17090>`__: MAINT: stats.rayleigh: enforce constraint on location
+* `#17105 <https://github.com/scipy/scipy/pull/17105>`__: FIX: special: use intended branching for \`lamv\` implementation
+* `#17166 <https://github.com/scipy/scipy/pull/17166>`__: MAINT: stats.rv_discrete.pmf: should be zero at non-integer argument
+* `#17176 <https://github.com/scipy/scipy/pull/17176>`__: REL: Prep for SciPy 1.9.3
+* `#17190 <https://github.com/scipy/scipy/pull/17190>`__: BUG: special: Fix two XSLOW test failures.
+* `#17193 <https://github.com/scipy/scipy/pull/17193>`__: MAINT: update meson.build to make it work on IBM i system
+* `#17200 <https://github.com/scipy/scipy/pull/17200>`__: BLD: fix issue with incomplete threads dependency handling
+* `#17204 <https://github.com/scipy/scipy/pull/17204>`__: Keepdims incorrectly documneted fix
+* `#17209 <https://github.com/scipy/scipy/pull/17209>`__: MAINT: Handle numpy's deprecation of accepting out-of-bound integers.
+* `#17210 <https://github.com/scipy/scipy/pull/17210>`__: BLD: fix invalid shebang for build helper script

meson.build

@@ -45,6 +45,14 @@ if m_dep.found()
   add_project_link_arguments('-lm', language : 'c')
 endif
 
+if host_machine.system() == 'os400'
+  # IBM i system, needed to avoid build errors - see gh-17193
+  add_project_arguments('-D__STDC_FORMAT_MACROS', language : 'cpp')
+  add_project_link_arguments('-Wl,-bnotextro', language : 'c')
+  add_project_link_arguments('-Wl,-bnotextro', language : 'cpp')
+  add_project_link_arguments('-Wl,-bnotextro', language : 'fortran')
+endif
+
 # Adding at project level causes many spurious -lgfortran flags.
 add_languages('fortran', native: false)
 ff = meson.get_compiler('fortran')

scipy/_build_utils/cythoner.py

@@ -1,4 +1,4 @@
-#!python3
+#!/usr/bin/env python3
 """ Scipy variant of Cython command
 
 Cython, as applied to single pyx file.

scipy/fft/_pocketfft/meson.build

@@ -1,12 +1,8 @@
-thread_dep = dependency('threads', required: false)
-
-# mingw-w64 does not implement pthread_pfork, needed by pocket_fft
-win_gcc = is_windows and meson.get_compiler('cpp').get_id() == 'gcc'
-
 pocketfft_threads = []
 fft_deps = [py3_dep]
 if is_windows
-  if win_gcc
+  if is_mingw
+    # mingw-w64 does not implement pthread_pfork, needed by pocket_fft
     # Disable threading completely, because of freezes using threading for
     # mingw-w64 gcc: https://github.com/mreineck/pocketfft/issues/1
     pocketfft_threads += ['-DPOCKETFFT_NO_MULTITHREADING']

scipy/interpolate/_bsplines.py

@@ -1241,9 +1241,17 @@ def make_interp_spline(x, y, k=3, t=None, bc_type=None, axis=0,
 
     y = np.moveaxis(y, axis, 0)    # now internally interp axis is zero
 
+    # sanity check the input
     if bc_type == 'periodic' and not np.allclose(y[0], y[-1], atol=1e-15):
         raise ValueError("First and last points does not match while "
                          "periodic case expected")
+    if x.size != y.shape[0]:
+        raise ValueError('Shapes of x {} and y {} are incompatible'
+                         .format(x.shape, y.shape))
+    if np.any(x[1:] == x[:-1]):
+        raise ValueError("Expect x to not have duplicates")
+    if x.ndim != 1 or np.any(x[1:] < x[:-1]):
+        raise ValueError("Expect x to be a 1D strictly increasing sequence.")
 
     # special-case k=0 right away
     if k == 0:
@@ -1289,17 +1297,10 @@ def make_interp_spline(x, y, k=3, t=None, bc_type=None, axis=0,
 
     t = _as_float_array(t, check_finite)
 
-    if x.ndim != 1 or np.any(x[1:] < x[:-1]):
-        raise ValueError("Expect x to be a 1-D sorted array_like.")
-    if np.any(x[1:] == x[:-1]):
-        raise ValueError("Expect x to not have duplicates")
     if k < 0:
         raise ValueError("Expect non-negative k.")
     if t.ndim != 1 or np.any(t[1:] < t[:-1]):
         raise ValueError("Expect t to be a 1-D sorted array_like.")
-    if x.size != y.shape[0]:
-        raise ValueError('Shapes of x {} and y {} are incompatible'
-                         .format(x.shape, y.shape))
     if t.size < x.size + k + 1:
         raise ValueError('Got %d knots, need at least %d.' %
                          (t.size, x.size + k + 1))

scipy/interpolate/_rgi.py

@@ -576,7 +576,7 @@ def interpn(points, values, xi, method="linear", bounds_error=True,
     if xi.shape[-1] != len(grid):
         raise ValueError("The requested sample points xi have dimension "
                          "%d, but this RegularGridInterpolator has "
-                         "dimension %d" % (xi.shape[1], len(grid)))
+                         "dimension %d" % (xi.shape[-1], len(grid)))
 
     if bounds_error:
         for i, p in enumerate(xi.T):

scipy/interpolate/fitpack/fpknot.f

@@ -21,7 +21,11 @@ recursive subroutine fpknot(x,m,t,n,fpint,nrdata,nrint,nest,
       real*8 an,am,fpmax
       integer ihalf,j,jbegin,jj,jk,jpoint,k,maxbeg,maxpt,
      * next,nrx,number
-c  ..
+
+c  note: do not initialize on the same line to avoid saving between calls
+      logical iserr
+      iserr = .TRUE.
+
       k = (n-nrint-1)/2
 c  search for knot interval t(number+k) <= x <= t(number+k+1) where
 c  fpint(number) is maximal on the condition that nrdata(number)
@@ -31,12 +35,15 @@ recursive subroutine fpknot(x,m,t,n,fpint,nrdata,nrint,nest,
       do 20 j=1,nrint
         jpoint = nrdata(j)
         if(fpmax.ge.fpint(j) .or. jpoint.eq.0) go to 10
+        iserr = .FALSE.
         fpmax = fpint(j)
         number = j
         maxpt = jpoint
         maxbeg = jbegin
   10    jbegin = jbegin+jpoint+1
   20  continue
+c  error condition detected, go to exit
+      if(iserr) go to 50
 c  let coincide the new knot t(number+k+1) with a data point x(nrx)
 c  inside the old knot interval t(number+k) <= x <= t(number+k+1).
       ihalf = maxpt/2+1
@@ -60,7 +67,7 @@ recursive subroutine fpknot(x,m,t,n,fpint,nrdata,nrint,nest,
       fpint(next) = fpmax*an/am
       jk = next+k
       t(jk) = x(nrx)
-      n = n+1
+  50  n = n+1
       nrint = nrint+1
       return
       end

scipy/interpolate/src/_fitpackmodule.c

@@ -1,8 +1,3 @@
-/*
-    Multipack project.
-    This file is generated by setmodules.py. Do not modify it.
- */
-
 #include <Python.h>
 #include "numpy/arrayobject.h"
 
@@ -15,6 +10,7 @@ static PyObject *fitpack_error;
 
 #define F_INT npy_int64
 #define F_INT_NPY NPY_INT64
+#define F_INT_MAX NPY_MAX_INT64
 
 #if NPY_BITSOF_SHORT == 64
 #define F_INT_PYFMT   "h"
@@ -32,9 +28,20 @@ static PyObject *fitpack_error;
 
 #else
 
-#define F_INT int
-#define F_INT_NPY NPY_INT
+#define F_INT npy_int32
+#define F_INT_NPY NPY_INT32
+#define F_INT_MAX NPY_MAX_INT32
+#if NPY_BITSOF_SHORT == 32
+#define F_INT_PYFMT   "h"
+#elif NPY_BITSOF_INT == 32
 #define F_INT_PYFMT   "i"
+#elif NPY_BITSOF_LONG == 32
+#define F_INT_PYFMT   "l"
+#else
+#error No compatible 32-bit integer size. \
+       Please contact NumPy maintainers and give detailed information about your \
+       compiler and platform
+#endif
 
 #endif
 
@@ -160,7 +167,7 @@ static PyObject *
 fitpack_bispev(PyObject *dummy, PyObject *args)
 {
     F_INT nx, ny, kx, ky, mx, my, lwrk, *iwrk, kwrk, ier, lwa, nux, nuy;
-    npy_intp mxy;
+    npy_intp int_max, mxy;
     double *tx, *ty, *c, *x, *y, *z, *wrk, *wa = NULL;
     PyArrayObject *ap_x = NULL, *ap_y = NULL, *ap_z = NULL, *ap_tx = NULL;
     PyArrayObject *ap_ty = NULL, *ap_c = NULL;
@@ -191,14 +198,19 @@ fitpack_bispev(PyObject *dummy, PyObject *args)
     ny = PyArray_DIMS(ap_ty)[0];
     mx = PyArray_DIMS(ap_x)[0];
     my = PyArray_DIMS(ap_y)[0];
-    mxy = mx*my;
-    if (my != 0 && mxy/my != mx) {
+    /* Limit is min of (largest array size, max of Fortran int) */
+    int_max = (F_INT_MAX < NPY_MAX_INTP) ? F_INT_MAX : NPY_MAX_INTP;
+    /* v = int_max/my is largest integer multiple of `my` such that
+       v * my <= int_max
+    */
+    if (my != 0 && int_max/my < mx) {
         /* Integer overflow */
         PyErr_Format(PyExc_RuntimeError,
                      "Cannot produce output of size %dx%d (size too large)",
                      mx, my);
         goto fail;
     }
+    mxy = (npy_intp)mx * (npy_intp)my;
     ap_z = (PyArrayObject *)PyArray_SimpleNew(1,&mxy,NPY_DOUBLE);
     if (ap_z == NULL) {
         goto fail;

scipy/interpolate/src/fitpack.pyf

@@ -685,7 +685,7 @@ static F_INT calc_regrid_lwrk(F_INT mx, F_INT my, F_INT kx, F_INT ky,
        real*8 optional :: r1
        real*8 optional,check(0.0<=s) :: s = 0.0
        integer intent(hide),depend(mu),check(nuest>=8) &
-            :: nuest = mu+6
+            :: nuest = mu+6+2
        integer intent(hide),depend(mv),check(nvest>=8) &
             :: nvest = mv+7
        integer intent(out) :: nu

scipy/interpolate/tests/test_bsplines.py

@@ -943,6 +943,29 @@ def test_linear(self):
         b = make_interp_spline(self.xx, self.yy, k=1, axis=-1)
         assert_allclose(b(self.xx), self.yy, atol=1e-14, rtol=1e-14)
 
+    @pytest.mark.parametrize('k', [0, 1, 2, 3])
+    def test_incompatible_x_y(self, k):
+        x = [0, 1, 2, 3, 4, 5]
+        y = [0, 1, 2, 3, 4, 5, 6, 7]
+        with assert_raises(ValueError, match="Shapes of x"):
+            make_interp_spline(x, y, k=k)
+
+    @pytest.mark.parametrize('k', [0, 1, 2, 3])
+    def test_broken_x(self, k):
+        x = [0, 1, 1, 2, 3, 4]      # duplicates
+        y = [0, 1, 2, 3, 4, 5]
+        with assert_raises(ValueError, match="x to not have duplicates"):
+            make_interp_spline(x, y, k=k)
+
+        x = [0, 2, 1, 3, 4, 5]      # unsorted
+        with assert_raises(ValueError, match="Expect x to be a 1D strictly"):
+            make_interp_spline(x, y, k=k)
+
+        x = [0, 1, 2, 3, 4, 5]
+        x = np.asarray(x).reshape((1, -1))     # 1D
+        with assert_raises(ValueError, match="Expect x to be a 1D strictly"):
+            make_interp_spline(x, y, k=k)
+
     def test_not_a_knot(self):
         for k in [3, 5]:
             b = make_interp_spline(self.xx, self.yy, k)

scipy/interpolate/tests/test_fitpack2.py

@@ -325,6 +325,33 @@ def test_array_like_input(self):
         assert_allclose(spl1([0.1, 0.5, 0.9, 0.99]),
                         spl2([0.1, 0.5, 0.9, 0.99]))
 
+    def test_fpknot_oob_crash(self):
+        # https://github.com/scipy/scipy/issues/3691
+        x = range(109)
+        y = [0., 0., 0., 0., 0., 10.9, 0., 11., 0.,
+             0., 0., 10.9, 0., 0., 0., 0., 0., 0.,
+             10.9, 0., 0., 0., 11., 0., 0., 0., 10.9,
+             0., 0., 0., 10.5, 0., 0., 0., 10.7, 0.,
+             0., 0., 11., 0., 0., 0., 0., 0., 0.,
+             10.9, 0., 0., 10.7, 0., 0., 0., 10.6, 0.,
+             0., 0., 10.5, 0., 0., 10.7, 0., 0., 10.5,
+             0., 0., 11.5, 0., 0., 0., 10.7, 0., 0.,
+             10.7, 0., 0., 10.9, 0., 0., 10.8, 0., 0.,
+             0., 10.7, 0., 0., 10.6, 0., 0., 0., 10.4,
+             0., 0., 10.6, 0., 0., 10.5, 0., 0., 0.,
+             10.7, 0., 0., 0., 10.4, 0., 0., 0., 10.8, 0.]
+        with suppress_warnings() as sup:
+            r = sup.record(
+                UserWarning,
+                r"""
+The maximal number of iterations maxit \(set to 20 by the program\)
+allowed for finding a smoothing spline with fp=s has been reached: s
+too small.
+There is an approximation returned but the corresponding weighted sum
+of squared residuals does not satisfy the condition abs\(fp-s\)/s < tol.""")
+            UnivariateSpline(x, y, k=1)
+            assert_equal(len(r), 1)
+
 
 class TestLSQBivariateSpline:
     # NOTE: The systems in this test class are rank-deficient
@@ -1209,6 +1236,20 @@ def test_negative_evaluation(self):
                         [-49.0625, -46.54315]])
         assert_array_almost_equal(data_interp, ans)
 
+    def test_pole_continuity_gh_14591(self):
+        # regression test for https://github.com/scipy/scipy/issues/14591
+        # with pole_continuty=(True, True), the internal work array size
+        # was too small, leading to a FITPACK data validation error.
+
+        # The reproducer in gh-14591 was using a NetCDF4 file with
+        # 361x507 arrays, so here we trivialize array sizes to a minimum
+        # which still demonstrates the issue.
+        u = np.arange(1, 10) * np.pi / 10
+        v = np.arange(1, 10) * np.pi / 10
+        r = np.zeros((9, 9))
+        for p in [(True, True), (True, False), (False, False)]:
+            RectSphereBivariateSpline(u, v, r, s=0, pole_continuity=p)
+
 
 def _numdiff_2d(func, x, y, dx=0, dy=0, eps=1e-8):
     if dx == 0 and dy == 0: