Properly retain types of Mapping subclasses #2325

ofek · 2021-02-07T16:39:36Z

Change Summary

This change prevents Mapping subclasses from always being coerced to dict

Related issue number

Resolves #2323
closes #1536

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

codecov · 2021-02-07T16:41:03Z

Codecov Report

Merging #2325 (651b160) into master (c8883e3) will decrease coverage by 0.13%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2325      +/-   ##
==========================================
- Coverage   99.90%   99.76%   -0.14%     
==========================================
  Files          25       25              
  Lines        5039     5062      +23     
  Branches     1031     1037       +6     
==========================================
+ Hits         5034     5050      +16     
- Misses          1        5       +4     
- Partials        4        7       +3

Impacted Files	Coverage Δ
pydantic/fields.py	`100.00% <100.00%> (ø)`
pydantic/main.py	`99.06% <100.00%> (-0.01%)`	⬇️
pydantic/schema.py	`100.00% <100.00%> (ø)`
pydantic/typing.py	`95.59% <0.00%> (-4.41%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8883e3...8b431df. Read the comment docs.

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

ofek · 2021-02-07T21:38:21Z

@PrettyWood I'm not sure how to fix the type checking errors

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

ofek · 2021-02-07T23:01:20Z

@PrettyWood The current job failure is due to https://pypi.org/project/cryptography/3.4.1/ emitting a warning in the test suite of FastAPI

PrettyWood · 2021-02-07T23:08:10Z

Yep can't do anything about that now but it should be handled rapidly on FastAPI side.
In the meantime you can add some tests to keep coverage at 100% :)

ofek · 2021-02-08T04:22:24Z

Done!

PrettyWood

@ofek We should still check the set type and try to coerce when needed.
For example

class M(BaseModel):
    x: Dict[str, int]

should coerce to dict. Right now there is no difference between Dict and Mapping...
I updated https://github.com/PrettyWood/pydantic/pull/70/files if you want

Note that I think we should probably have a proper validator for Dict (at least) before Mapping to avoid any performance impact on normal dictionaries (like it's done for Tuple or List, which are treated before Sequence)

ofek · 2021-02-08T15:07:59Z

Updated based on your branch

PrettyWood

I still reckon we should at least have SHAPE_DICT (extra could be SHAPE_DEFAULTDICT, SHAPE_COUNTER...) before SHAPE_MAPPING to have no perf impact.
Or at least if we go with it we should have a benchmark to see the impact

pydantic/fields.py

ofek · 2021-02-10T16:32:52Z

@PrettyWood Addressed. I'm not familiar with the concept of shapes here. Can you explain?

PrettyWood · 2021-02-10T16:46:10Z

@ofek pydantic is fast because a lot is done in the metaclass. You can have a look at ModelField._type_analysis(), which is run in ModelMetaclass when ModelField.infer() is called.
In _type_analysis a lot is done to set the right attributes to each ModelField (type, shape, validators...)
And the shape is used to call the right default validators to speed up the parsing

ofek · 2021-02-10T17:19:18Z

I just tried that but many tests are failing now

ofek · 2021-02-10T18:27:17Z

should I revert the last commit for now?

ofek · 2021-02-11T16:21:51Z

I'm still trying but I don't understand what's wrong 🙁

ofek · 2021-02-12T17:28:38Z

@PrettyWood Could you please take a look when you get a chance?

PrettyWood · 2021-02-12T17:30:37Z

I'll have a look tonight or tomorrow 👍

ofek · 2021-02-12T17:34:09Z

Thank you, feel free to push directly to my branch!

PrettyWood · 2021-02-13T09:52:22Z

Hi @ofek
Had a quick look and got all tests working in local PrettyWood#70
I have some guests coming around soon but I think you should be able to work it out without a problem
I think you still need to add some tests to showcase a random mapping getting coerced into dict when Dict[...] is set.
Hope it helps

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

ofek · 2021-02-13T15:00:06Z

@PrettyWood Done! Added that test too

@samuelcolvin

I let @samuelcolvin have a look. Thanks!

samuelcolvin

otherwise this is looking good.

pydantic/fields.py

tests/test_main.py

ofek · 2021-02-14T12:46:20Z

@samuelcolvin Thanks, addressed!

ofek · 2021-02-16T05:23:01Z

Done this look alright now?

ofek · 2021-02-17T21:28:50Z

@samuelcolvin Just let me know if I have to do anything else! FYI, after this is merged and released, every Datadog Agent integration will exclusively use pydantic for user configuration 🙂

samuelcolvin · 2021-02-25T17:28:40Z

This is great. Thank you.

@samuelcolvin Just let me know if I have to do anything else! FYI, after this is merged and released, every Datadog Agent integration will exclusively use pydantic for user configuration 🙂

I don't see pydantic as a dependency, what am I missing?

PrettyWood · 2021-02-25T17:38:56Z

@samuelcolvin DataDog/integrations-core#8675

ofek · 2021-03-22T17:41:45Z

DataDog/integrations-core#8675 is merged and I've started the syncing of every integration: https://github.com/DataDog/integrations-core/pulls 😄

ofek · 2021-03-29T19:52:18Z

https://datadoghq.dev/integrations-core/meta/config-models/

Properly retain types of Mapping subclasses

600b20f

Create 2325-ofek.md

ac69d33

This was referenced Feb 7, 2021

External dict-like type gets mapped to dict (unwanted behaviour) #2311

Closed

Mapping types are coerced to dict #2323

Closed

ofek and others added 2 commits February 7, 2021 16:11

update with feedback

2cf67b8

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

satisfy mypy?

fbb26ec

Update fields.py

947ae57

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

ofek added 2 commits February 7, 2021 20:17

show uncovered line numbers

95b68b4

fix coverage

432d0e2

PrettyWood previously requested changes Feb 8, 2021

View reviewed changes

update

a472dc1

PrettyWood reviewed Feb 9, 2021

View reviewed changes

pydantic/fields.py Outdated Show resolved Hide resolved

pydantic/fields.py Outdated Show resolved Hide resolved

pydantic/fields.py Outdated Show resolved Hide resolved

ofek added 2 commits February 10, 2021 11:47

address feedback

35f1071

try

abbbb60

Merge branch 'master' into retain-mapping-types

2b77312

update

59bdac7

Co-Authored-By: Eric Jolibois <eric.jolibois@toucantoco.com>

rename test

c5c7e19

samuelcolvin reviewed Feb 14, 2021

View reviewed changes

pydantic/fields.py Outdated Show resolved Hide resolved

tests/test_main.py Outdated Show resolved Hide resolved

tests/test_main.py Outdated Show resolved Hide resolved

address feedback

a078d50

Merge branch 'master' into retain-mapping-types

8b431df

samuelcolvin merged commit 4ddf4f1 into pydantic:master Feb 25, 2021

ofek deleted the retain-mapping-types branch February 25, 2021 17:39

This was referenced Mar 6, 2021

Bump pydantic from 1.7.3 to 1.8.1 in /backend Vaarna/vaarna#139

Merged

Bump pydantic from 1.6.1 to 1.8.1 in /emmet-core rkingsbury/emmet#25

Closed

dependabot bot mentioned this pull request Mar 17, 2021

Bump pydantic[dotenv] from 1.7.3 to 1.8.1 enchant97/web-portal#1

Merged

JacobHayes mentioned this pull request Mar 19, 2023

Excluding fields when copying a model replaces dict subclasses with dict #5225

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Properly retain types of Mapping subclasses #2325

Properly retain types of Mapping subclasses #2325

ofek commented Feb 7, 2021 •

edited by PrettyWood

codecov bot commented Feb 7, 2021 •

edited

ofek commented Feb 7, 2021

ofek commented Feb 7, 2021

PrettyWood commented Feb 7, 2021

ofek commented Feb 8, 2021

PrettyWood left a comment •

edited

ofek commented Feb 8, 2021

PrettyWood left a comment

ofek commented Feb 10, 2021

PrettyWood commented Feb 10, 2021

ofek commented Feb 10, 2021

ofek commented Feb 10, 2021

ofek commented Feb 11, 2021

ofek commented Feb 12, 2021

PrettyWood commented Feb 12, 2021

ofek commented Feb 12, 2021

PrettyWood commented Feb 13, 2021

ofek commented Feb 13, 2021

samuelcolvin left a comment

ofek commented Feb 14, 2021

ofek commented Feb 16, 2021

ofek commented Feb 17, 2021

samuelcolvin commented Feb 25, 2021

PrettyWood commented Feb 25, 2021

ofek commented Mar 22, 2021

ofek commented Mar 29, 2021

Properly retain types of Mapping subclasses #2325

Properly retain types of Mapping subclasses #2325

Conversation

ofek commented Feb 7, 2021 • edited by PrettyWood

Change Summary

Related issue number

Checklist

codecov bot commented Feb 7, 2021 • edited

Codecov Report

ofek commented Feb 7, 2021

ofek commented Feb 7, 2021

PrettyWood commented Feb 7, 2021

ofek commented Feb 8, 2021

PrettyWood left a comment • edited

Choose a reason for hiding this comment

ofek commented Feb 8, 2021

PrettyWood left a comment

Choose a reason for hiding this comment

ofek commented Feb 10, 2021

PrettyWood commented Feb 10, 2021

ofek commented Feb 10, 2021

ofek commented Feb 10, 2021

ofek commented Feb 11, 2021

ofek commented Feb 12, 2021

PrettyWood commented Feb 12, 2021

ofek commented Feb 12, 2021

PrettyWood commented Feb 13, 2021

ofek commented Feb 13, 2021

samuelcolvin left a comment

Choose a reason for hiding this comment

ofek commented Feb 14, 2021

ofek commented Feb 16, 2021

ofek commented Feb 17, 2021

samuelcolvin commented Feb 25, 2021

PrettyWood commented Feb 25, 2021

ofek commented Mar 22, 2021

ofek commented Mar 29, 2021

ofek commented Feb 7, 2021 •

edited by PrettyWood

codecov bot commented Feb 7, 2021 •

edited

PrettyWood left a comment •

edited