numpy.ndarray.clip segfaults on big-endian dtype (Trac #1588)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/1588 on 2010-08-20 by trac user bdkern, assigned to unknown.

import numpy
numpy.zeros((1,), dtype='>i2').clip(min=0)

segfaults on three systems tried: Ubuntu / Python 2.5.2 / Numpy 1.1.1, Red Hat / py 2.6.4 / np 1.3.0, WinXP / py 2.5.4 / np 1.3.0. This same setup also segfaults for dtypes '>f4', '>f8', etc.

[numpy-gitbot]

@fengy-research wrote on 2012-05-08

I am confirming this is still and issue with 1.6.1 with IPython, on python 2.7.3

won't seg fault:
.clip(min=0, max=1), with ipython
.clip(min=0), with python (I believe there may still be corrupted data)
will seg fault:
.clip(min=0), with ipython.

I made a gdb attaching to the ipython process and obtained a backtrace:

0x00002aaab026d5df in _update_descr_and_dimensions (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0,
flags=1287, obj=0x12c9d2e0) at numpy/core/src/multiarray/ctors.c:265
265 Py_INCREF(*des);
(gdb) bt
#0 0x00002aaab026d5df in _update_descr_and_dimensions (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0,
flags=1287, obj=0x12c9d2e0) at numpy/core/src/multiarray/ctors.c:265
#1 PyArray_NewFromDescr (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0, flags=1287, obj=0x12c9d2e0)
at numpy/core/src/multiarray/ctors.c:961
#2 0x00002aaab02a999d in PyArray_Newshape (self=0x12c9d2e0, newdims=, order=NPY_CORDER) at numpy/core/src/multiarray/shape.c:301
#3 0x00002aaab02af2d4 in PyArray_Ravel (a=0x0, order=NPY_CORDER) at numpy/core/src/multiarray/shape.c:845
#4 0x00002aaab02af4b9 in array_ravel (self=0x12c9d2e0, args=, kwds=) at numpy/core/src/multiarray/methods.c:2030
#5 0x000000000049b574 in call_function (f=0x12e47d50, throwflag=) at Python/ceval.c:4012
#6 PyEval_EvalFrameEx (f=0x12e47d50, throwflag=) at Python/ceval.c:2665
#7 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f957b0, globals=, locals=, args=0x2, argcount=1,
kws=0x12e47cd8, kwcount=0, defs=0x11f93728, defcount=1, closure=0x0) at Python/ceval.c:3252
#8 0x000000000049aeb5 in call_function (f=0x12e47ae0, throwflag=) at Python/ceval.c:4108
#9 PyEval_EvalFrameEx (f=0x12e47ae0, throwflag=) at Python/ceval.c:2665
#10 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f8f2b0, globals=, locals=, args=0x5, argcount=6,
kws=0x12e47aa8, kwcount=0, defs=0x11f9e260, defcount=2, closure=0x0) at Python/ceval.c:3252
#11 0x000000000049aeb5 in call_function (f=0x12e478b0, throwflag=) at Python/ceval.c:4108
#12 PyEval_EvalFrameEx (f=0x12e478b0, throwflag=) at Python/ceval.c:2665
#13 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f8f430, globals=, locals=, args=0x7, argcount=6,
kws=0x12e47878, kwcount=0, defs=0x11f98548, defcount=6, closure=0x0) at Python/ceval.c:3252
#14 0x000000000049aeb5 in call_function (f=0x12e47680, throwflag=) at Python/ceval.c:4108
#15 PyEval_EvalFrameEx (f=0x12e47680, throwflag=) at Python/ceval.c:2665
#16 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11eeddb0, globals=, locals=, args=0x4, argcount=1, kws=0x0,
kwcount=0, defs=0x2b9bb36971a8, defcount=3, closure=0x0) at Python/ceval.c:3252
#17 0x00000000004fb1bf in function_call (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Objects/funcobject.c:526
#18 0x00000000004186ed in PyObject_Call (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Objects/abstract.c:2522
#19 0x00000000004945a6 in PyEval_CallObjectWithKeywords (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Python/ceval.c:3881
#20 0x00002aaab025cedd in array_repr (self=) at numpy/core/src/multiarray/arrayobject.c:435
#21 0x000000000044d57a in PyObject_Repr (v=0x12c9d2e0) at Objects/object.c:381
#22 0x000000000049b736 in call_function (f=0x12e47460, throwflag=) at Python/ceval.c:4000
#23 PyEval_EvalFrameEx (f=0x12e47460, throwflag=) at Python/ceval.c:2665
#24 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x1197e930, globals=, locals=, args=0x12e46e08, argcount=3,
kws=0x12e46e20, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#25 0x000000000049aeb5 in call_function (f=0x12e46c50, throwflag=) at Python/ceval.c:4108
#26 PyEval_EvalFrameEx (f=0x12e46c50, throwflag=) at Python/ceval.c:2665

[numpy-gitbot]

@fengy-research wrote on 2012-05-08

Another investigation running gdb with a breakpoint on PyArray_Clip shows the function runs properly.

And there is no way for PyArray_Clip to fall into the PyArray_Ravel path. Who is calling PyArray_Ravel?

[numpy-gitbot]

@fengy-research wrote on 2012-05-08

More strangely,

with ipython -c 'import numpy; a=numpy.array([11214], dtype=">f4"); a.clip(min=0);'

The code always crashes.

But with the interactive ipython -pylab, the code sometimes does crash sometimes does not.

[numpy-gitbot]

@charris wrote on 2012-05-11

Still present in 1.7.0.dev-bb162a2.

>>> a = numpy.zeros((2,), dtype='>i2').clip(min=0)
>>> a + a
Segmentation fault (core dumped)

I'm guessing a reference counting bug.

[numpy-gitbot]

Milestone changed to NumPy 1.7 by @charris on 2012-05-11

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

Here is the stacktrace as of master 5c944b9:

https://gist.github.com/3548633

for the following code:

import numpy
a = numpy.zeros((2,), dtype='>i2').clip(min=0)
print "OK 1"
x = a + a
print "OK 2"
y = str(x)
print "OK 3"
print y
print "OK 4"

It fails on the line "y = str(x)".

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

When compiled using:

CFLAGS="-g -ggdb -O0" python setup.py install

the full stacktrace is:

https://gist.github.com/3548988

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

It segfaults on the line:

        return descr->f->getitem(data, base);

Where descr->f is ok, but descr->f->getitem points to a memory (not NULL) that cannot be accessed. A full output from valgrind: https://gist.github.com/3549063

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

The following patch seems to fix it (now there are leaks):

diff --git a/numpy/core/src/multiarray/calculation.c b/numpy/core/src/multiarray/calculation.c
index 618a871..1dd4a18 100644
--- a/numpy/core/src/multiarray/calculation.c
+++ b/numpy/core/src/multiarray/calculation.c
@@ -931,7 +931,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     }
     if (max != NULL) {
         newdescr = PyArray_DescrFromObject(max, indescr);
-        Py_XDECREF(indescr);
+        //Py_XDECREF(indescr);
         if (newdescr == NULL) {
             return NULL;
         }
@@ -959,12 +959,12 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
         indescr = PyArray_DESCR(self);
         Py_INCREF(indescr);
     }
-    Py_DECREF(newdescr);
+    //Py_DECREF(newdescr);

     if (!PyDataType_ISNOTSWAPPED(indescr)) {
         PyArray_Descr *descr2;
         descr2 = PyArray_DescrNewByteorder(indescr, '=');
-        Py_DECREF(indescr);
+        //Py_DECREF(indescr);
         if (descr2 == NULL) {
             goto fail;
         }
@@ -981,7 +981,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     }
     else {
         /* Side-effect of PyArray_FromAny */
-        Py_DECREF(indescr);
+        //Py_DECREF(indescr);
     }

     /*
@@ -998,14 +998,14 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
             zero = PyInt_FromLong(0);
             cmp = PyObject_RichCompareBool(min, zero, Py_LT);
             if (cmp == -1) {
-                Py_DECREF(zero);
+                //Py_DECREF(zero);
                 goto fail;
             }
             if (cmp == 1) {
                 min = zero;
             }
             else {
-                Py_DECREF(zero);
+                //Py_DECREF(zero);
                 Py_INCREF(min);
             }
         }
@@ -1017,7 +1017,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
         Py_INCREF(indescr);
         mina = (PyArrayObject *)PyArray_FromAny(min, indescr, 0, 0,
                                  NPY_ARRAY_DEFAULT, NULL);
-        Py_DECREF(min);
+        //Py_DECREF(min);
         if (mina == NULL) {
             goto fail;
         }
@@ -1147,11 +1147,11 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     func(PyArray_DATA(newin), PyArray_SIZE(newin), min_data, max_data, PyArray_DATA(newout));

     /* Clean up temporary variables */
-    Py_XDECREF(mina);
-    Py_XDECREF(maxa);
-    Py_DECREF(newin);
+    //Py_XDECREF(mina);
+    //Py_XDECREF(maxa);
+    //Py_DECREF(newin);
     /* Copy back into out if out was not already a nice array. */
-    Py_DECREF(newout);
+    //Py_DECREF(newout);
     return (PyObject *)out;

  fail:

So the problem is right here somewhere.

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

I am going to post the rest of the issues here:

#398

Due to trac unreliability.

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

This is fixed by: #405

[numpy-gitbot]

trac user ocertik wrote on 2012-08-31

PR is merged.