Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: avoid dict coercion for subclasses #3138

Merged

Conversation

PrettyWood
Copy link
Member

@PrettyWood PrettyWood commented Aug 26, 2021

Change Summary

When using a subclass of dict as type of a field, the shape of the field was still considered SHAPE_DICT. When a value was validated, it was hence coerced to dict here
Now we set the shape to SHAPE_MAPPING in those cases and keep target class

Related issue number

closes #3122

Checklist

  • Unit tests for the changes exist
  • Tests pass on CI and coverage remains at 100%
  • Documentation reflects the changes where applicable
  • changes/<pull request or issue id>-<github username>.md file added describing change
    (see changes/README.md for details)
  • My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

@PrettyWood
Copy link
Member Author

please review

@@ -630,7 +630,7 @@ def _type_analysis(self) -> None: # noqa: C901 (ignore complexity)
self.key_field = self._create_sub_type(get_args(self.type_)[0], 'key_' + self.name, for_keys=True)
self.type_ = get_args(self.type_)[1]
self.shape = SHAPE_DEFAULTDICT
elif issubclass(origin, Dict):
elif origin is dict or origin is Dict:
Copy link
Contributor

@JacobHayes JacobHayes Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted bad suggestion

Maybe:

        elif issubclass(origin, (dict, Dict)):

?

further investigation

Hmm the origin is dict or origin is Dict does fix the conversion to dict, but I still get the AttributeError: name for the subclass in the example in #3122. 🤔 Maybe that's a separate problem though.

...

One additional piece of info: it looks like type(Base.__fields__["b"].outer_type_) is <class 'types.GenericAlias'>, but type(Sub.__fields__["b"].outer_type_) is <class 'dict'>. 🤔

It looks like maybe .outer_type_ is getting replaced with some other dictionary instance somehow in subclass creation?

(Pdb) m.__fields__["b"].outer_type_.keys()
dict_keys([4336358208, 4340970304, 4341193536, 4341045632, 4341212736, 4340612832, 4341193344, 4341212480, 4341239232, 4341200384, 4341200448, 4340970688, 4341193856])
(Pdb) m.__fields__["b"].outer_type_.values()
*** AttributeError: name

Update:

This PR does seem to fix the frozendict[str, int] -> dict conversion for the base class (yay, thanks)! However, there is a bug with the upstream frozendict package (prompted by the deepcopy of __fields__ on subclass creation):

>>> from frozendict import frozendict
>>> from copy import copy, deepcopy
>>>
>>> deepcopy(frozendict)
<class 'frozendict.core.frozendict'>
>>> deepcopy(frozendict[str, int])
{}

I'll open a separate ticket there.

Copy link
Member Author

@PrettyWood PrettyWood Sep 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @JacobHayes
Thanks for the review
So yes what didn't work before is

m = Base(a=frozendict(a=1), b=frozendict(b='1'))
assert isinstance(m.a, frozendict)
assert isinstance(m.b, frozendict)
assert m.b['b'] == 1  # coerced

This is now resolved because the frozendict is not coerced in dict.
The 'name' issue is something else indeed.

EDIT: Great that you found the root cause! I let you see on frozendict side or monkeypatch it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case you're interested, this escalated to a cPython bug with types.GenericAlias: https://bugs.python.org/issue45167

@JacobHayes
Copy link
Contributor

@PrettyWood do you know if this might make it into the next release? 🙏

@PrettyWood
Copy link
Member Author

It should

Copy link
Member

@samuelcolvin samuelcolvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

otherwise looks good, just not clear what happens in these cases


class Model(BaseModel):
a: MyDict
b: MyDict[str, int]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
b: MyDict[str, int]
b: MyDict[str, int]
c: Dict[str, int]
d: Mapping[str, int]

What happens to these types with the input being a MyDict?

Copy link
Member Author

@PrettyWood PrettyWood Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for c coercion is done (as currently expected with pydantic) and for d no coercion is done. I added them in the test

@samuelcolvin
Copy link
Member

please update

@PrettyWood
Copy link
Member Author

Please review

@samuelcolvin
Copy link
Member

awesome, thank you.

@PrettyWood PrettyWood deleted the avoid-dict-coercion-subclasses branch December 8, 2021 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AttributeError: name with frozendict[..., ...] and subclass (maybe Mapping+GenericAlias issue?)
3 participants