[BUG] Generators after being cythonized cannot be inspect as generators

[Tiarles]

Describe the bug

Generators that are cythonized cannot be identified as generators when inspect.isgeneratorfunction is used.

Code to reproduce the behaviour:

The code below is used in the package that will be cythonized:

# File src/test_gens/impl.pyx
# Code to be cythonized
def _my_generator():
    value = 6
    print("my_generator:", value)
    yield value
    value = value + 6
    print("my_generator:", value)

and then used by the test

# File test.py
import inspect
from test_gens import my_generator

for i in my_generator():
    print("test:", i)

print("inspect.isgeneratorfunction(main.my_generator):", inspect.isgeneratorfunction(my_generator))

# inspect.isgeneratorfunction(main.my_generator): False

Expected behaviour

Running the same test with the src/test_gens/impl.pyx code in a .py file the return is True, as expected.

OS

Windows

Python version

3.8.4, 3.9.6, 3.10.4, 3.11.4

Cython version

3.0.0

Additional context

No response

[Tiarles]

I initiate the discussion of this issue here.

[da-woods]

It works for me on Cython 3.0.9 with Python 3.11.5.

Have you set the directive # cython: binding = False somewhere? That turns off a lot of introspection and means that inspect.isgeneratorfunction won't work on them. It's on by default though.

[Tiarles]

Fun fact: I found on the repository a comment that was not made by me but never called to my attention:

ext_modules = cythonize(
    ["**/*.pyx"],
    compiler_directives={
        "binding": True,  # so inspect function works properly on cythonized functions
        "language_level": language_level,
    },  # avoid cython warnings
)

So yes, explicitly the biding is set as True.

I look for the line # cython: binding = False or just cython: binding but no place else we define that.

I can try with the most recent Cython version and let you know the outcome.

[Tiarles]

No changes with cython==3.0.9. Let me know if a more explanatory example is needed.

[da-woods]

To clarify, what I tested was

# gen.py
def _my_generator():
    value = 6
    print("my_generator:", value)
    yield value
    value = value + 6
    print("my_generator:", value)

and then tested it with:

# test_gen.py
import gen
import inspect
import os

print(os.path.split(gen.__file__)[1])
print(gen._my_generator)
print(gen._my_generator.__code__.co_flags)
print(inspect.isgeneratorfunction(gen._my_generator))

That gives

gen.py
<function _my_generator at 0x7fa38359cd60>
35
True

Then if I do cythonize -if gen.py and run it again I get

gen.cpython-311-x86_64-linux-gnu.so
<cyfunction _my_generator at 0x7f286ca69e50>
35
True

I don't think your example that you show in the issue is quite what you actually run - for example the module and function names don't match.

[Tiarles]

The only step that I miss explaining is that the my_generator function is built as a package. Cythonized using the follow lines on setup.py file:

ext_modules = cythonize(
    ["**/*.pyx"],
    compiler_directives={
        "binding": True,  # so inspect function works properly on cythonized functions
        "language_level": language_level,
    },  # avoid cython warnings
)

Then, the module is installed and imports the function from the cythonized wheel. I can create a small repository with the example and attach it to our discussion with a tox file for execution.

[Tiarles]

I managed to pack nicely in a repository (cy_wheel_gens_tests).

[da-woods]

The issue is your wrapper in __init__.py which doesn't have anything to do with Cython. Here's a reduced version of your example (no Cython necessary):

def _my_generator():
    yield 1

def my_generator():
    return _my_generator()

import inspect

print(inspect.isgeneratorfunction(my_generator))

This prints False because of the wrapper.

[Tiarles]

Aha, I see. I will make a better example because the issue that we are having is on pluggy verification of pytest hooks.

I will investigate more and if persists I will open a new ticket. Thank you for your time.

[Tiarles]

Please, if is possible can you try to take a look at example again ? In the same repository here. I upadted the README.md with the updates made on implementation and on the test.

[da-woods]

Works for me on 3.10 and 3.11:

====================================================== 1 passed in 0.01s ======================================================
  py38: FAIL code 1 (5.00=setup[0.04]+cmd[0.54,2.93,1.04,0.45] seconds)
  py39: FAIL code 1 (4.75=setup[0.01]+cmd[0.51,2.87,0.91,0.44] seconds)
  py310: OK (4.61=setup[0.01]+cmd[0.46,2.81,0.94,0.40] seconds)
  py311: OK (4.04=setup[0.01]+cmd[0.45,2.47,0.89,0.21] seconds)

On 3.8, 3.9 inspect.isfunction requires return isinstance(object, types.FunctionType) which we cannot meet.

The changes in inspect that make it work right came in Python 3.10.6 and 3.11.0. (python/cpython#94050)

There's no way for us to make it work on earlier versions of Python.

[Tiarles]

Understood. Anyway, thank you once more for your time.

[da-woods]

Your other option would be to monkey-patch inspect. I think we do it ourselves inside runtests.py (

cython/runtests.py

Line 140 in 07553a5

def patch_inspect_isfunction():

)

At one point Cython did this specifically for isgenerator in some circumstances. That wouldn't have helped you specifically, although the basic idea could have been used. I think it got removed - possibly semi-mistakenly mainly because it wasn't tested. I don't think Cython should return to doing that in Cython itself though.