Compatibility with Python 3.11 #907

frenzymadness · 2022-01-14T06:42:31Z

Hello.

In Fedora, we are again trying to prepare the new Python 3.11 soon and during the rebuild of our packages, we have discovered that some tests of attrs package fail:

We are building attrs 21.4.0 together with Python 3.11.0~a3 (changelog)

=================================== FAILURES ===================================
_______________________ test_init_subclass_vanilla[True] _______________________

slots = True

    @pytest.mark.parametrize("slots", [True, False])
    def test_init_subclass_vanilla(slots):
        """
        `super().__init_subclass__` can be used if the subclass is not an attrs
        class both with dict and slotted classes.
        """
    
        @attr.s(slots=slots)
        class Base:
            def __init_subclass__(cls, param, **kw):
                super().__init_subclass__(**kw)
                cls.param = param
    
>       class Vanilla(Base, param="foo"):

tests/test_init_subclass.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'tests.test_init_subclass.test_init_subclass_vanilla.<locals>.Vanilla'>
param = 'foo', kw = {}

    def __init_subclass__(cls, param, **kw):
>       super().__init_subclass__(**kw)
E       TypeError: super(type, obj): obj must be an instance or subtype of type

tests/test_init_subclass.py:24: TypeError
_____________ TestClosureCellRewriting.test_closure_cell_rewriting _____________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec6da50>

    def test_closure_cell_rewriting(self):
        """
        Slotted classes support proper closure cell rewriting.
    
        This affects features like `__class__` and the no-arg super().
        """
        non_slot_instance = C1(x=1, y="test")
        slot_instance = C1Slots(x=1, y="test")
    
        assert non_slot_instance.my_class() is C1
>       assert slot_instance.my_class() is C1Slots
E       AssertionError: assert <class 'tests.test_slots.C1Slots'> is C1Slots
E        +  where <class 'tests.test_slots.C1Slots'> = <bound method C1Slots.my_class of C1Slots(x=1, y='test')>()
E        +    where <bound method C1Slots.my_class of C1Slots(x=1, y='test')> = C1Slots(x=1, y='test').my_class

tests/test_slots.py:424: AssertionError
__________________ TestClosureCellRewriting.test_inheritance ___________________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec658d0>

    def test_inheritance(self):
        """
        Slotted classes support proper closure cell rewriting when inheriting.
    
        This affects features like `__class__` and the no-arg super().
        """
    
        @attr.s
        class C2(C1):
            def my_subclass(self):
                return __class__
    
        @attr.s
        class C2Slots(C1Slots):
            def my_subclass(self):
                return __class__
    
        non_slot_instance = C2(x=1, y="test")
        slot_instance = C2Slots(x=1, y="test")
    
        assert non_slot_instance.my_class() is C1
>       assert slot_instance.my_class() is C1Slots
E       AssertionError: assert <class 'tests.test_slots.C1Slots'> is C1Slots
E        +  where <class 'tests.test_slots.C1Slots'> = <bound method C1Slots.my_class of C2Slots(x=1, y='test')>()
E        +    where <bound method C1Slots.my_class of C2Slots(x=1, y='test')> = C2Slots(x=1, y='test').my_class

tests/test_slots.py:451: AssertionError
________________ TestClosureCellRewriting.test_cls_static[True] ________________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec71c90>
slots = True

    @pytest.mark.parametrize("slots", [True, False])
    def test_cls_static(self, slots):
        """
        Slotted classes support proper closure cell rewriting for class- and
        static methods.
        """
        # Python can reuse closure cells, so we create new classes just for
        # this test.
    
        @attr.s(slots=slots)
        class C:
            @classmethod
            def clsmethod(cls):
                return __class__
    
>       assert C.clsmethod() is C
    
        @attr.s(slots=slots)
E       AssertionError: assert <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'> is <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>
E        +  where <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'> = <bound method TestClosureCellRewriting.test_cls_static.<locals>.C.clsmethod of <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>>()
E        +    where <bound method TestClosureCellRewriting.test_cls_static.<locals>.C.clsmethod of <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>> = <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>.clsmethod

tests/test_slots.py:475: AssertionError
____________________ test_slots_super_property_get_shurtcut ____________________

    @pytest.mark.skipif(PY2, reason="shortcut super() is PY3-only.")
    def test_slots_super_property_get_shurtcut():
        """
        On Python 3, the `super()` shortcut is allowed.
        """
    
        @attr.s(slots=True)
        class A(object):
            x = attr.ib()
    
            @property
            def f(self):
                return self.x
    
        @attr.s(slots=True)
        class B(A):
            @property
            def f(self):
                return super().f ** 2
    
>       assert B(11).f == 121

tests/test_slots.py:739: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = B(x=11)

    @property
    def f(self):
>       return super().f ** 2
E       TypeError: super(type, obj): obj must be an instance or subtype of type

tests/test_slots.py:737: TypeError
=============================== warnings summary ===============================
../../BUILDROOT/python-attrs-21.4.0-1.fc36.x86_64/usr/lib/python3.11/site-packages/attr/_make.py:918
tests/test_init_subclass.py::test_init_subclass_vanilla[True]
tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True]
tests/test_slots.py::test_slots_empty_cell
tests/test_slots.py::test_slots_super_property_get
tests/test_slots.py::test_slots_super_property_get_shurtcut
  /builddir/build/BUILDROOT/python-attrs-21.4.0-1.fc36.x86_64/usr/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
    set_closure_cell(cell, cls)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/test_3rd_party.py:14: could not import 'cloudpickle': No module named 'cloudpickle'
SKIPPED [1] tests/test_make.py:469: No old-style classes in Py3
SKIPPED [1] tests/test_make.py:1002: PY2-specific keyword-only error behavior
SKIPPED [1] tests/test_make.py:1967: Needs to be only caught on Python 2.
SKIPPED [1] tests/test_make.py:2333: Pre-3.10 only.
SKIPPED [1] tests/test_pyright.py:26: Requires pyright.
SKIPPED [1] tests/test_slots.py:485: can't break CodeType.replace() via monkeypatch
SKIPPED [1] tests/test_slots.py:532: slots without weakref_slot should only work on PyPy
XFAIL tests/test_setattr.py::TestSetAttr::test_slotted_confused
FAILED tests/test_init_subclass.py::test_init_subclass_vanilla[True] - TypeEr...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_closure_cell_rewriting
FAILED tests/test_slots.py::TestClosureCellRewriting::test_inheritance - Asse...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True]
FAILED tests/test_slots.py::test_slots_super_property_get_shurtcut - TypeErro...
====== 5 failed, 1166 passed, 8 skipped, 1 xfailed, 6 warnings in 19.65s =======

The text was updated successfully, but these errors were encountered:

Tinche · 2022-01-14T11:34:56Z

@frenzymadness should we raise this on the Python side? They're breaking code that works on previous versions. The code is very black magic though.

frenzymadness · 2022-01-14T15:45:12Z

I haven't had time to investigate it more. If you think those are bugs in Python, then yes, open issues for them. From a quick look, I don't see anything related to this in the changelog.

Tinche · 2022-01-16T22:49:23Z

Link to BPO: https://bugs.python.org/issue46404

frenzymadness · 2022-01-17T05:42:01Z

Thank you very much for the upstream report. It's interesting reading.

frenzymadness · 2022-03-16T08:26:03Z

As I mentioned in the bug in Python upstream, I'd like to make this situation more stable which means to implement the fix you have here for dataclasses and test it in Python so every future change in behavior of __closure__ cells is detected and investigated.

I'm able to reproduce the problem with property:

@dataclasses.dataclass(slots=True)
class A:
    @property
    def prop(self):
        return __class__

a = A()
print(a.prop)

and also with inheritance and arg-less call to super:

@dataclasses.dataclass(slots=True)
class A:
    pass

@dataclasses.dataclass(slots=True)
class B(A):
    def test(self):
        super()

b = B()
b.test()

but the same problem should appear in classmethod and staticmethod but I'm not able to reproduce those. I'm able to find the __closure__ cell of the new class but it already points to the new class instead of the old one. Is it possible that this has changed in Python 3.11? I'm testing this code:

import dataclasses

@dataclasses.dataclass(slots=True)
class A:
    @classmethod
    def class_method(self):
        return __class__

a = A()
a.class_method()

And it works fine in both 3.11 and 3.10 without any fixes. slots have been added to dataclasses in 3.10 so I cannot check this with older versions.

You seem to understand the problem very well so I thought you might know how to reproduce the third use case.

frenzymadness · 2022-03-25T12:19:13Z

@vstinner could you please help me here. Are classmethods and staticmethods something we should care about here or not?

vstinner · 2022-03-28T13:52:02Z

I created https://bugs.python.org/issue47143 "Add functools.copy_class() which updates closures".

Tinche · 2022-03-28T13:59:02Z

@vstinner That would be great!

frenzymadness · 2022-05-04T07:41:56Z

Unfortunately, it seems that it's not easily doable to implement such general-purpose function in Python, see the reasons mentioned in the last comment: python/cpython#91299 (comment)

We'll probably have to keep the code in attrs with all the branches for older and newer Pythons.

mgorny · 2022-05-09T18:32:04Z

With 3.11.0b1, I see even more test failures:

FAILED tests/test_3rd_party.py::TestCloudpickleCompat::test_repr - IndexError: tuple index out of range
FAILED tests/test_annotations.py::TestAnnotations::test_auto_attribs[True] - AssertionError: assert {'a': <class ...ist[int], ...} =...
FAILED tests/test_annotations.py::TestAnnotations::test_auto_attribs[False] - AssertionError: assert {'a': <class ...ist[int], ...} ...
FAILED tests/test_annotations.py::TestAnnotations::test_annotations_strings[True] - AssertionError: assert {'a': <class ...ist[int],...
FAILED tests/test_annotations.py::TestAnnotations::test_annotations_strings[False] - AssertionError: assert {'a': <class ...ist[int]...
FAILED tests/test_init_subclass.py::test_init_subclass_vanilla[True] - TypeError: super(type, obj): obj must be an instance or subty...
FAILED tests/test_make.py::TestAutoDetect::test_detects_setstate_getstate[True] - AssertionError: assert None is <built-in method __...
FAILED tests/test_make.py::TestAutoDetect::test_detects_setstate_getstate[False] - AssertionError: assert None is <built-in method _...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_closure_cell_rewriting - AssertionError: assert <class 'tests.test_slots....
FAILED tests/test_slots.py::TestClosureCellRewriting::test_inheritance - AssertionError: assert <class 'tests.test_slots.C1Slots'> i...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True] - AssertionError: assert <class 'tests.test_slots.TestCl...
FAILED tests/test_slots.py::TestPickle::test_no_getstate_setstate_for_dict_classes - AssertionError: assert None is <built-in method...
FAILED tests/test_slots.py::TestPickle::test_no_getstate_setstate_if_option_false - AssertionError: assert None is <built-in method ...
FAILED tests/test_slots.py::test_slots_super_property_get - TypeError: super(type, obj): obj must be an instance or subtype of type
FAILED tests/test_slots.py::test_slots_super_property_get_shortcut - TypeError: super(type, obj): obj must be an instance or subtype...

Full log:
py311.txt

frenzymadness · 2022-06-20T13:41:54Z

Just linking another PR: #969

hynek · 2022-06-24T09:55:54Z

fixed by #969

vstinner · 2022-06-24T10:19:39Z

Great, thanks!

@SnoopJ

But still support older attrs versions on older Python versions Thank-you-to: James Gerity @SnoopJ Reference: python-attrs/attrs#907 Reference: python-attrs/attrs#969 Reference: #3199 (comment) Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

hynek added the Bug label Jan 14, 2022

Tinche mentioned this issue Jan 16, 2022

Fix set_closure_cell on 3.11 #910

Closed

vstinner mentioned this issue May 3, 2022

Add types.copy_class() which updates closures python/cpython#91299

Closed

mgorny mentioned this issue Jun 3, 2022

Test failures with Python 3.11: TypeError: super(type, obj): obj must be an instance or subtype of type saltstack/pytest-shell-utilities#28

Closed

hynek closed this as completed Jun 24, 2022

felicianotech mentioned this issue Oct 31, 2022

Python v3.11.0 & python-attr CircleCI-Public/cimg-python#168

Closed

pombredanne mentioned this issue Jan 16, 2023

Support python 3.11 nexB/scancode-toolkit#3199

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with Python 3.11 #907

Compatibility with Python 3.11 #907

frenzymadness commented Jan 14, 2022

Tinche commented Jan 14, 2022

frenzymadness commented Jan 14, 2022

Tinche commented Jan 16, 2022

frenzymadness commented Jan 17, 2022

frenzymadness commented Mar 16, 2022

frenzymadness commented Mar 25, 2022

vstinner commented Mar 28, 2022

Tinche commented Mar 28, 2022

frenzymadness commented May 4, 2022

mgorny commented May 9, 2022

frenzymadness commented Jun 20, 2022

hynek commented Jun 24, 2022

vstinner commented Jun 24, 2022

Compatibility with Python 3.11 #907

Compatibility with Python 3.11 #907

Comments

frenzymadness commented Jan 14, 2022

Tinche commented Jan 14, 2022

frenzymadness commented Jan 14, 2022

Tinche commented Jan 16, 2022

frenzymadness commented Jan 17, 2022

frenzymadness commented Mar 16, 2022

frenzymadness commented Mar 25, 2022

vstinner commented Mar 28, 2022

Tinche commented Mar 28, 2022

frenzymadness commented May 4, 2022

mgorny commented May 9, 2022

frenzymadness commented Jun 20, 2022

hynek commented Jun 24, 2022

vstinner commented Jun 24, 2022