Excluding fields when copying a model replaces dict subclasses with dict

[JacobHayes]

Initial Checks

• I have searched GitHub for a duplicate issue and I'm sure this is something new
• I have searched Google & StackOverflow for a solution and couldn't find anything
• I have read and followed the docs and still think this is a bug
• I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description

When a model has a field with a dict "subclass" (eg: defaultdict, Box, etc), you can normally create and .copy() the model retaining the dict subclass (implemented in #2325 and #3138 - not sure of the functional difference between them 🤔).

However, if you pass an include/exclude to .copy() or have a Field(..., exclude=True), then the call to BaseModel._get_value() will convert the subclass to a direct dict:

pydantic/pydantic/main.py

Lines 485 to 500 in 38f0b4e

	if isinstance(v, dict):
	return {
	k_: cls._get_value(
	v_,
	to_dict=to_dict,
	by_alias=by_alias,
	exclude_unset=exclude_unset,
	exclude_defaults=exclude_defaults,
	include=value_include and value_include.for_element(k_),
	exclude=value_exclude and value_exclude.for_element(k_),
	exclude_none=exclude_none,
	)
	for k_, v_ in v.items()
	if (not value_exclude or not value_exclude.is_excluded(k_))
	and (not value_include or value_include.is_included(k_))
	}

This results in random issues depending on the subclass and usage (in my example, I was using a frozen Model+Box to preserve hash-ability and all sorts of other things).

Example Code

from collections import defaultdict

from pydantic import BaseModel


class M(BaseModel):
    id: int
    data: defaultdict[str, int]


m = M(id=5, data=defaultdict(int, {"a": 5}))

print(m)                       # M(id=5, data=defaultdict(<class 'int'>, {'a': 5}))
print(m.copy())                # M(id=5, data=defaultdict(<class 'int'>, {'a': 5}))
print(m.copy(exclude={"id"}))  # M(data={'a': 5})

Python, Pydantic & OS Version

pydantic version: 1.10.6
            pydantic compiled: True
                 install path: /Users/jacobhayes/src/github.com/artigraph/artigraph/.direnv/python-3.11.0/lib/python3.11/site-packages/pydantic
               python version: 3.11.0 (main, Nov 28 2022, 13:26:47) [Clang 13.1.6 (clang-1316.0.21.2.5)]
                     platform: macOS-13.2.1-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

Affected Components

• Compatibility between releases
• Data validation/parsing
• Data serialization - .dict() and .json()
• JSON Schema
• Dataclasses
• Model Config
• Field Types - adding or changing a particular data type
• Function validation decorator
• Generic Models
• Other Model behaviour - construct(), pickling, private attributes, ORM mode
• Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[samuelcolvin]

Humm, I doubt we can fix this in v1, I'm not even sure how we can fix this in V2, I think we should only allow include/exclude on known types - models, dataclasses, dict, list, tuple - in particular I there think we will need an error if you try to include/exclude on a type we can't help on.

[sydney-runkle]

We’re no longer actively developing Pydantic V1 (although it will continue to receive security fixes for the next year or so), so we’re closing issues solely related to V1. If you really think this shouldn’t be closed, please comment or create a new issue 🚀.