BUG: Cython and Py 3.10 have create issues when debugging

[christian-steinmeyer]

Describe the issue:

I'm experiencing some inconsistent behavior for np.dtype and np.iinfo. From within some code I cannot share here, I run the snippet below which fails with a TypeError (details also below). If I simply execute a main in which I call this function directly, it works however. Since I have a call stack of about 10 levels in the error case, I don't know, where exactly the difference is caused. If someone knows a way of easily and quickly isolating this issue, I'm happy to apply it.

The same code does not yield an error, if run in Python 3.8.5 and numpy 1.19.5. It does occur in python 3.10 and numpy versions 1.21.2, 1.22.0, 1.22.1, and 1.22.2.

Reproduce the code example:

import numpy as np

def my_func(dtype):
    dtype = np.dtype(dtype)
    iinfo = np.iinfo(dtype)  # <-- fails here

...
 my_func("uint_8")

Error message:

Traceback (most recent call last):
  File "/Users/<USER>/.virtualenvs/<PROJECT>/lib/python3.10/site-packages/numpy/core/getlimits.py", line 516, in __init__
    self.dtype = numeric.dtype(int_type)
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'numpy.dtype[uint8]'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  ...
    iinfo = np.iinfo(dtype)
  File "/Users/<USER>/.virtualenvs/<PROJECT>/lib/python3.10/site-packages/numpy/core/getlimits.py", line 518, in __init__
    self.dtype = numeric.dtype(type(int_type))
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'numpy._DTypeMeta'

NumPy/Python version information:

1.21.2 3.10.0 (default, Dec 7 2021, 09:05:11) [Clang 12.0.5 (clang-1205.0.22.9)]

[seberg]

Seems weird, but two questions:

1. Do you use Cython anywhere (or pandas/random numbers that use Cython)?
2. Does this by any chance only happen with profiling debugging enabled?

There is some weird issue with 3.10 + Cython + trancing (i.e. Pythons profiling/debugging hook). I think the Cython re-release fixes it, but unless your project is the Cython component that might be hard to use.

And this looks a lot like that weird issue (pandas runs into it if you debug/profile) due to callbacks from pandas to NumPy (python): pandas-dev/pandas#41935

[christian-steinmeyer]

Thanks for the quick response!
You are right, it seems to only happen while debugging. I don't use any cython myself, but it's definitely possible, that any of the third party code I use does. I applied the approach suggested in the linked pandas issue and indeed found a change in numpy.core.numeric.dtype from

numeric.dtype=<class 'numpy.dtype'>
type(numeric.dtype)=<class 'numpy._DTypeMeta'>

to

numeric.dtype=<class 'numpy.uint64'>
type(numeric.dtype)=<class 'type'>

around some random sampling using numpy's SFC64 BitGenerator.

[seberg]

Thanks, so the underlying issue seems in Cython due to Python changes and fixed in Cythons the upcoming release. I guess workarounds in NumPy may be possible for the specific symptoms, right now I would hope that Cython can fix and we can regenerate random with that. But I am not sure how much more we may see of this, because Python 3.10 is probably not used quite that much yet.

As soon as there is a Cython fix available, we need to pin against that maybe...

[christian-steinmeyer]

I'd be happy to help with testing any new versions :)

[seberg]

No tests needed, the only question is probably whether we should make sure to include the workaround from the Cython issue in the next NumPy release (there is an option to disable the faulty code-paths, although that might add some slight overheads).

[christian-steinmeyer]

I see. That I would leave to the numpy devs.

[seberg]

The quick fix (until the next cython release) is to do:

Defining the cflag CYTHON_FAST_PYCALL=0 seems to fix it though (at the cost of some speed I assume).

This may slow down some things in random slightly. The other option is to wait for the next release or to just use the alpha version.

[christian-steinmeyer]

This doesn't seem to work for me, but since it's already schedule for the 1.22.4 I'm just gonna wait for the update :)

[seberg]

This doesn't seem to work for me,

I am a bit surprised, so to note that this is a C define I believe, and not an environment variable. So use would probably look like: CFLAGS="-DCYTHON_FAST_PYCALL=0" as an environment variable (I have not double checked it though).

(EDIT: Also, this is at compile time of course)

[christian-steinmeyer]

I see, sorry. Misunderstood that one. In that case it might work, but I didn't try it.

[charris]

@seberg What do you think we should do for 1.,22.4? Looks like Cython isn't going to fix it.

[seberg]

I don't think there is anything we can do besides using CFLAGS="-DCYTHON_FAST_PYCALL=0" during compilation for the cython modules. That is likely fine for np.random, but it may also be that many who run into this will need to ask e.g. pandas also to do it (and pandas may well notice it more).
I have no idea how he performance impact of this would be.

[charris]

Should we do that for the Wheels? I think that can be done for the docker images through env_vars, not sure how to do that for Windows and Mac. Maybe do a 1.22.4 release for testing?

[mattip]

We could measure the performance change when using CFLAGS="-DCYTHON_FAST_PYCALL=0" for cython-compiled code.

Maybe we should try to have a discussion with @scoder about what the path forward should be. CPython 3.11 is just around the corner: do we continue with Cython0.29? Do we encourage use of Cython3?

[charris]

Do we encourage use of Cython3?

Hmm. We will need to do that eventually, I guess the question is, is it ready.

[seberg]

I would be happy if we just tried out the alpha even. But I am unsure we should do so in a point release.

Looking at the performance impact for an example function that use a lot of Python calls back to NumPy (or simliar) would maybe be best to get an idea if just disabling it is even noticable in practice.

[scoder]

Do we encourage use of Cython3?

Hmm. We will need to do that eventually, I guess the question is, is it ready.

The largest remaining open question for Cython 3 is really what we do with cython/cython#4280 (letting C functions propagate exceptions by default). That's a tough call. We have been trying hard to keep the transition easy for users and to avoid breaking existing code wherever possible. But that's a very tempting property, and Cython 3 is the right time to make that change.

[seberg]

Thanks for fixing it in cython/cython#4735 on the cython side. So it would be nice to publish the next wheels in a way that ensures this is used.

[Mereep]

I am running in the same issue and it really troubles me right now. This is not the first time these days, kicking in the debugger changed the behaviour of the program itself. Had a similar thing going on with the mysql-connector which was eating exceptions when being run in the debugger.

This makes me question why and how a debugger changes the behaviour of the program? Somehow this seems broken.

[charris]

Hmm. I'm going to delay 1.22.4 a while in the hope that Cython makes a new release soon so that we can fix this.

[SimoneD89]

I just wanted to report that cython 0.29.29 was released yesterday

[charris]

Cython 0.29.30 is out and merged for the upcoming 1.22.4, so this should be fixed. If the problem persists feel free to reopen.

[davidgilbertson]

@charris I don't have Cython installed and I get the error TypeError: 'NoneType' object is not callable (only when debugging, in IntelliJ). Is this a related issue, or should I create a new issue?

It still occurs with 1.22.4.

Reference Jetbrains issue: https://youtrack.jetbrains.com/issue/PY-52137

pd.show_versions()
INSTALLED VERSIONS
------------------
commit           : 4bfe3d07b4858144c219b9346329027024102ab6
python           : 3.10.2.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.22000
machine          : AMD64
processor        : Intel64 Family 6 Model 151 Stepping 2, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_Australia.1252
pandas           : 1.4.2
numpy            : 1.22.4
pytz             : 2022.1
dateutil         : 2.8.2
pip              : 22.0.4
setuptools       : 62.3.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.8.0
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 3.1.2
IPython          : 8.3.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : 1.3.4
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
markupsafe       : 2.1.1
matplotlib       : 3.5.2
numba            : None
numexpr          : 2.8.1
odfpy            : None
openpyxl         : 3.0.10
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.8.1
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : 0.8.9
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None

[NeanderThaler]

Have the same issue.
Python 3.10.4
numpy 1.22.4
pandas 1.4.2

Still get TypeError: 'NoneType' object is not callable
Only when debugging (in PyCharm)

Out of the box i hadn't cython installed.
After trying to fix the issue I installed Cython 0.29.30 and reinstalled all libraries. But nothing changed, still get the errror.

[NeanderThaler]

To add a (maybe important) observation. The bug seems to be fixed when calling things from the main-loop. But when using threads (I'm using Qt Threads), the bug still persists

[seberg]

The bug is related with tracing (debugging/profiling) and not threads I think. In any case, NumPy being fixed is not really much help because in most cases this is triggered by pandas or someone else.
You need to update the triggering package (something that uses cython) as well, and not all of them may have a new release yet.

[NeanderThaler]

Yeah, you're right. It wasn't related to the threads. But to the order how stuff is loaded

The following code causes the error:

import pandas as pd
pd.DataFrame([])
from scipy import signal

The following code is fine:

from scipy import signal
import pandas as pd
pd.DataFrame([])

Just the order of when scipy is imported is different.

The combination of libraries makes this really difficult. At a different point the application fails when triggering sklearn. But when only running the code that uses sklearn in unittest, I can debug.
I can play around with this and try different combinations of how libraries are loaded and used. Some work, most don't. But in the case where they don't it's always numpy.core.getlimits that causes the problems.

[seberg]

@NeanderThaler if you think this is the problem, I would suggest you just solve it by applying patience: wait a week or two and upgrade all of the packages that may be affected.

Beyond that, the only thing you can do easily is avoid "tracing". If sys.gettrace() returns something and you are on Python 3.10, you will have a problem but maybe that only happens for some weirder CI (profiling, debugging) so that you can just avoid it.
(That said, some IDEs may use, probably partial, tracing in their python console.)