TypeError when registering ForwardRefs in python 3.10.2

[JWCook]

• cattrs version: 1.10.0
• Python version: 3.10.2
• Operating System: Ubuntu 20.04.3

Description

This is related to #201, but probably worth making a separate issue for, since it affects one of the (previously working) workarounds mentioned in that issue. Feel free to close this if you feel this is redundant with #201 or #94.

I have a converter that registers a ForwardRef. In python 3.6.x through 3.10.1, this has worked as intended. As of python 3.10.2, this is now throwing a TypeError:

TypeError: Invalid first argument to `register()`. ForwardRef('MyClass') is not a class.

Strangely, I don't see anything in the python 3.10.2 changelog that would indicate changed behavior with ForwardRef or register().

What I Did

Minimal example:

from attr import define, field
from cattr import Converter  # Note: same behavior with GenConverter
from typing import ForwardRef, List

@define()
class MyClass:
    history: List['MyClass'] = field(factory=list)

converter = Converter()
converter.register_structure_hook(
    ForwardRef('MyClass'), lambda obj, _: converter.structure(obj, MyClass)
)

Traceback:

TypeError                                 Traceback (most recent call last)
Input In [17], in <module>
----> 1 converter.register_structure_hook(
      2     ForwardRef("MyClass"), (lambda obj, _: converter.structure(obj, MyClass))
      3 )

File ~/.virtualenvs/rc-310/lib/python3.10/site-packages/cattr/converters.py:269, in Converter.register_structure_hook(self, cl, func)
    267     self._structure_func.clear_cache()
    268 else:
--> 269     self._structure_func.register_cls_list([(cl, func)])

File ~/.virtualenvs/rc-310/lib/python3.10/site-packages/cattr/dispatch.py:57, in MultiStrategyDispatch.register_cls_list(self, cls_and_handler, direct)
     55         self._direct_dispatch[cls] = handler
     56     else:
---> 57         self._single_dispatch.register(cls, handler)
     58         self.clear_direct()
     59 self.dispatch.cache_clear()

File ~/.pyenv/versions/3.10.2/lib/python3.10/functools.py:856, in singledispatch.<locals>.register(cls, func)
    854 else:
    855     if func is not None:
--> 856         raise TypeError(
    857             f"Invalid first argument to `register()`. "
    858             f"{cls!r} is not a class."
    859         )
    860     ann = getattr(cls, '__annotations__', {})
    861     if not ann:

TypeError: Invalid first argument to `register()`. ForwardRef('MyClass') is not a class.

[JWCook]

Note that one of the other solutions posted in #201 still works in python 3.10.2: #201 (comment)

So the above example would become:

from attr import define, field
from cattr import GenConverter
from typing import ForwardRef, List

@define()
class MyClass:
    history: List['MyClass'] = field(factory=list)

converter = GenConverter()
converter .register_structure_hook_func(
    lambda t: t.__class__ is typing.ForwardRef,
    lambda v, t: converter.structure(v, t.__forward_value__),
)

So this may be a non-issue, but probably still useful for someone else out there googling for the same error.

[aha79]

I think the error is due to this commit (a change in functools.py), which was a fix due to bpo-46032.

It seems that the newly added check _is_valid_dispatch_type() returns false and thus we get the TypeError. (I cannot run the example as only Python 10.2 for macos is out).

It is a bit hard to follow the logic, but ForwardRef("MyClass") technically is an instance and not a type (which may explain the behavior).

However, I do not understand is why the solution in your last comment works.

Also what the change to singledispatch means for cattrs. I have the feeling that this will bite us again.

[JWCook]

Good to know, thanks for tracking that down. It looks like this may be an unintended side effect of that bugfix rather than an intentional change, then.

It is a bit hard to follow the logic, but ForwardRef("MyClass") technically is an instance and not a type (which may explain the behavior).

True, but an instance of ForwardRef also looks like a class:

>>> from typing import ForwardRef
>>> type(ForwardRef("MyClass"))
<class 'typing.ForwardRef'>

However, I do not understand is why the solution in your last comment works.

Tinche would be able to give a more thorough explanation, but I believe it works because Converter.register_structure_hook_func() takes a different path than register_structure_hook(), via FunctionDispatch.dispatch():

cattrs/src/cattr/dispatch.py

Lines 95 to 131 in 22b24c2

	@attr.s(slots=True)
	class FunctionDispatch:
	"""
	FunctionDispatch is similar to functools.singledispatch, but
	instead dispatches based on functions that take the type of the
	first argument in the method, and return True or False.
	objects that help determine dispatch should be instantiated objects.
	"""
	_handler_pairs: list = attr.ib(factory=list)
	def register(
	self, can_handle: Callable[[Any], bool], func, is_generator=False
	):
	self._handler_pairs.insert(0, (can_handle, func, is_generator))
	def dispatch(self, typ):
	"""
	returns the appropriate handler, for the object passed.
	"""
	for can_handle, handler, is_generator in self._handler_pairs:
	# can handle could raise an exception here
	# such as issubclass being called on an instance.
	# it's easier to just ignore that case.
	try:
	ch = can_handle(typ)
	except Exception:
	continue
	if ch:
	if is_generator:
	return handler(typ)
	else:
	return handler
	raise StructureHandlerNotFoundError(
	f"unable to find handler for {typ}", type_=typ
	)

We're using the function we provide (in this case testing t.__class__ is typing.ForwardRef) instead of the checks in functools.singledispatch.register(), so _is_valid_dispatch_type() is never called on a ForwardRef instance.

[Tinche]

Yeah, as you folks found out, singledispatch is very inadequate for a bunch of use cases using more abstract types (i.e. not actual classes). That's why after checking singledispatch cattrs has a list of predicates that it'll check in order, and that's what you're enabling with register_structure_hook_func. I'm actually surprised ForwardRefs worked before with singledispatch.

So the predicate approach would be preferred, yeah.

[ermshiperete]

Same bug occurs with Python 3.9.10 (on Debian/Ubuntu).

[ermshiperete]

See also the bug in requests-cache: requests-cache/requests-cache#501

[alekseiloginov]

Hi, any timeline to fix this issue for python 3.9.10? Thanks

[ermshiperete]

Hi, any timeline to fix this issue for python 3.10.2? Thanks

This is fixed in requests-cache 0.9.1

[alekseiloginov]

Thanks, but I'm curious more about cattrs, since I'm getting it from cattrs 1.10.0:

  File "/opt/brew/lib/python3.9/site-packages/cattr/converters.py", line 269, in register_structure_hook
    self._structure_func.register_cls_list([(cl, func)])
  File "/opt/brew/lib/python3.9/site-packages/cattr/dispatch.py", line 57, in register_cls_list
    self._single_dispatch.register(cls, handler)
  File "/opt/brew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/functools.py", line 855, in register
    raise TypeError(
TypeError: Invalid first argument to `register()`. ForwardRef('AccessToken') is not a class.

[Tinche]

@alekseiloginov cattrs has no "official" support for ForwardRefs yet (or has ever had it). The register_structure_hook approach worked while singledispatch supported it, but since https://bugs.python.org/issue46032 was released it no longer supports it.

We might build ForwardRef support into cattrs in a special way (there's some interest) in the future, but until then simply switch from using register_structure_hook to using register_structure_hook_func (see #206 (comment)). This isn't a hack, this is how cattrs supports all hooks that cannot work via singledispatch (there are many).

[ex-nerd]

Note that one of the other solutions posted in #201 still works in python 3.10.2: #201 (comment)

I wasn't able to get my slightly different situation to work with python==3.10.4, cattrs==22.1.0 (MacOS, pyenv). It runs, but fails to structure things properly (just leaves the nested property as a regular dict instead of my class). However, one of the other solutions from #201 (comment) works great: just put the whole thing in quotes.

from attr import define, field

@define()
class MyClass:
    history: "Optional[List[MyClass]]" = field(default=None)