Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: change pydantic dataclass decorator #2557

Merged
merged 51 commits into from Aug 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
665b5c9
refactor: rewrite the whole pydantic dataclass logic
PrettyWood Mar 21, 2021
b723e59
test: add tests for issue 2162
PrettyWood Mar 27, 2021
914f3bb
test: add tests for issue 2383
PrettyWood Mar 27, 2021
8706598
test: add tests for issue 2398
PrettyWood Mar 27, 2021
2e15ef5
test: add tests for issue 2424
PrettyWood Mar 27, 2021
f3d68ce
test: add tests for issue 2541
PrettyWood Mar 28, 2021
95f1d15
test: add tests for issue 2555
PrettyWood Mar 27, 2021
f5a5756
refactor: polish
PrettyWood Mar 28, 2021
9ae2e30
change default and support 3.6
PrettyWood Mar 28, 2021
d7b2b27
fix coverage
PrettyWood Mar 28, 2021
64dbc56
fix mypy and text
PrettyWood Mar 28, 2021
e86f8bb
typos
PrettyWood Mar 28, 2021
4086193
test: add tests for issue 2594
PrettyWood Mar 29, 2021
dc717fe
fix: forward doc for schema description
PrettyWood Apr 11, 2021
9799d80
add change
PrettyWood Apr 12, 2021
e7cbee1
chore: small changes from review
PrettyWood May 16, 2021
538c7b3
refactor: avoid extra __pydantic_run_validation__ parameter
PrettyWood May 16, 2021
3874661
Merge branch 'master' into refactor/dataclass-decorator
PrettyWood May 16, 2021
9174d7a
small tweaks
PrettyWood May 16, 2021
adda021
remove wrapper
PrettyWood May 16, 2021
bf0f191
support 3.6
PrettyWood May 16, 2021
c3aa37d
fix: mypy
PrettyWood May 16, 2021
461fa1b
rewrite doc
PrettyWood May 16, 2021
77f4d6b
add docs
PrettyWood May 16, 2021
263859c
wrapper is removed now
PrettyWood May 16, 2021
066f23c
a bit more docs
PrettyWood May 16, 2021
432ae04
Merge branch 'master' into refactor/dataclass-decorator
PrettyWood Sep 4, 2021
1926e85
Merge branch 'master' into refactor/dataclass-decorator
PrettyWood Sep 6, 2021
e3a8f87
code review
PrettyWood Sep 6, 2021
80da159
faster dict update
PrettyWood Sep 6, 2021
bde50bb
add test for issue 3162
PrettyWood Sep 6, 2021
eee2e26
add test for issue 3011
PrettyWood Sep 6, 2021
d101420
feat: add `Config.post_init_after_validation`
PrettyWood Sep 6, 2021
04f7ea0
allow config via dict
PrettyWood Sep 7, 2021
8e24502
fix cython and TypedDict
PrettyWood Sep 7, 2021
84ec7b5
Merge branch 'master' into refactor/dataclass-decorator
PrettyWood Dec 11, 2021
a8237d8
chore: typo
PrettyWood Dec 11, 2021
5c5d152
move `compiled` in `version.py`
PrettyWood Dec 11, 2021
6715d4c
refactor: switch from `Config.post_init_after_validation` to \'post_i…
PrettyWood Dec 11, 2021
c6d8201
add dataclass isinstance support
PrettyWood Dec 11, 2021
6914c04
avoid multi paragraphs in change file
PrettyWood Dec 11, 2021
30b58f3
feat: support `Config.extra`
PrettyWood Dec 11, 2021
0079d93
Merge branch 'master' into refactor/dataclass-decorator
PrettyWood Dec 20, 2021
fb77d0d
refactor: simplify a bit code
PrettyWood Dec 20, 2021
1a60ab2
refactor: avoid creating useless functions
PrettyWood Dec 20, 2021
ffecfc2
refactor: simplify `is_builtin_dataclass`
PrettyWood Dec 20, 2021
3a6ddf1
support extra in post_init
PrettyWood Dec 20, 2021
5907f82
docs: add warning on config extra
PrettyWood Dec 20, 2021
571609d
Merge branch 'master' into PrettyWood-refactor/dataclass-decorator
samuelcolvin Aug 4, 2022
4810457
fix #3713 compatibility
samuelcolvin Aug 4, 2022
0c7d578
update docs
samuelcolvin Aug 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 9 additions & 0 deletions changes/2557-PrettyWood.md
@@ -0,0 +1,9 @@
Refactor the whole _pydantic_ `dataclass` decorator to really act like its standard lib equivalent.
It hence keeps `__eq__`, `__hash__`, ... and makes comparison with its non-validated version possible.
It also fixes usage of `frozen` dataclasses in fields and usage of `default_factory` in nested dataclasses.
The support of `Config.extra` has been added.
Finally, config customization directly via a `dict` is now possible.
<br/><br/>
**BREAKING CHANGES**
- The `compiled` boolean (whether _pydantic_ is compiled with cython) has been moved from `main.py` to `version.py`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this really count as breaking? The import from main will still work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from pydantic.main import compiled will fail. If it doesn't matter let's remove those 2 lines

- Now that `Config.extra` is supported, `dataclass` ignores by default extra arguments (like `BaseModel`)
26 changes: 26 additions & 0 deletions docs/examples/dataclasses_config.py
@@ -0,0 +1,26 @@
from pydantic import ConfigDict
from pydantic.dataclasses import dataclass


# Option 1 - use directly a dict
# Note: `mypy` will still raise typo error
@dataclass(config=dict(validate_assignment=True))
class MyDataclass1:
a: int


# Option 2 - use `ConfigDict`
# (same as before at runtime since it's a `TypedDict` but with intellisense)
@dataclass(config=ConfigDict(validate_assignment=True))
class MyDataclass2:
a: int


# Option 3 - use a `Config` class like for a `BaseModel`
class Config:
validate_assignment = True


@dataclass(config=Config)
class MyDataclass3:
a: int
30 changes: 30 additions & 0 deletions docs/examples/dataclasses_stdlib_run_validation.py
@@ -0,0 +1,30 @@
import dataclasses

from pydantic import ValidationError
from pydantic.dataclasses import dataclass as pydantic_dataclass, set_validation


@dataclasses.dataclass
class User:
id: int
name: str


# Enhance stdlib dataclass
pydantic_dataclass(User)


user1 = User(id='whatever', name='I want')

# validate data of `user1`
try:
user1.__pydantic_validate_values__()
except ValidationError as e:
print(e)

# Enforce validation
try:
with set_validation(User, True):
User(id='whatever', name='I want')
except ValidationError as e:
print(e)
20 changes: 16 additions & 4 deletions docs/examples/dataclasses_stdlib_to_pydantic.py
Expand Up @@ -16,20 +16,32 @@ class File(Meta):
filename: str


File = pydantic.dataclasses.dataclass(File)
# `ValidatedFile` will be a proxy around `File`
ValidatedFile = pydantic.dataclasses.dataclass(File)

file = File(
# the original dataclass is the `__dataclass__` attribute
assert ValidatedFile.__dataclass__ is File


validated_file = ValidatedFile(
filename=b'thefilename',
modified_date='2020-01-01T00:00',
seen_count='7',
)
print(file)
print(validated_file)

try:
File(
ValidatedFile(
filename=['not', 'a', 'string'],
modified_date=None,
seen_count=3,
)
except pydantic.ValidationError as e:
print(e)

# `File` is not altered and still does no validation by default
print(File(
filename=['not', 'a', 'string'],
modified_date=None,
seen_count=3,
))
34 changes: 33 additions & 1 deletion docs/usage/dataclasses.md
Expand Up @@ -18,7 +18,7 @@ You can use all the standard _pydantic_ field types, and the resulting dataclass
created by the standard library `dataclass` decorator.

The underlying model and its schema can be accessed through `__pydantic_model__`.
Also, fields that require a `default_factory` can be specified by a `dataclasses.field`.
Also, fields that require a `default_factory` can be specified by either a `pydantic.Field` or a `dataclasses.field`.

```py
{!.tmp_examples/dataclasses_default_schema.py!}
Expand All @@ -34,6 +34,20 @@ keyword argument `config` which has the same meaning as [Config](model_config.md
For more information about combining validators with dataclasses, see
[dataclass validators](validators.md#dataclass-validators).

## Dataclass Config

If you want to modify the `Config` like you would with a `BaseModel`, you have three options:

```py
{!.tmp_examples/dataclasses_config.py!}
```

!!! warning
After v1.10, _pydantic_ dataclasses support `Config.extra` but some default behaviour of stdlib dataclasses
may prevail. For example, when `print`ing a _pydantic_ dataclass with allowed extra fields, it will still
use the `__str__` method of stdlib dataclass and show only the required fields.
This may be improved further in the future.

## Nested dataclasses

Nested dataclasses are supported both in dataclasses and normal models.
Expand All @@ -51,12 +65,25 @@ Dataclasses attributes can be populated by tuples, dictionaries or instances of

Stdlib dataclasses (nested or not) can be easily converted into _pydantic_ dataclasses by just decorating
them with `pydantic.dataclasses.dataclass`.
_Pydantic_ will enhance the given stdlib dataclass but won't alter the default behaviour (i.e. without validation).
It will instead create a wrapper around it to trigger validation that will act like a plain proxy.
The stdlib dataclass can still be accessed via the `__dataclass__` attribute (see example below).

```py
{!.tmp_examples/dataclasses_stdlib_to_pydantic.py!}
```
_(This script is complete, it should run "as is")_

### Choose when to trigger validation

As soon as your stdlib dataclass has been decorated with _pydantic_ dataclass decorator, magic methods have been
added to validate input data. If you want, you can still keep using your dataclass and choose when to trigger it.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. I wrote some code which wraps standard lib dataclasses using pydantic dataclasses when I want to call validation to achieve the same thing. Would love to see it implemented in the pydantic lib itself so we can allow users to set up their model however they like (i.e. without having to pass everything to the init func) and parse/validate when they are done.


```py
{!.tmp_examples/dataclasses_stdlib_run_validation.py!}
```
_(This script is complete, it should run "as is")_

### Inherit from stdlib dataclasses

Stdlib dataclasses (nested or not) can also be inherited and _pydantic_ will automatically validate
Expand Down Expand Up @@ -95,6 +122,11 @@ When you initialize a dataclass, it is possible to execute code *after* validati
with the help of `__post_init_post_parse__`. This is not the same as `__post_init__`, which executes
code *before* validation.

!!! tip
If you use a stdlib `dataclass`, you may only have `__post_init__` available and wish the validation to
be done before. In this case you can set `Config.post_init_call = 'after_validation'`


```py
{!.tmp_examples/dataclasses_post_init_post_parse.py!}
```
Expand Down
4 changes: 4 additions & 0 deletions docs/usage/model_config.md
Expand Up @@ -118,6 +118,10 @@ not be included in the model schemas. **Note**: this means that attributes on th
**`smart_union`**
: whether _pydantic_ should try to check all types inside `Union` to prevent undesired coercion; see [the dedicated section](#smart-union)

**`post_init_call`**
: whether stdlib dataclasses `__post_init__` should be run before (default behaviour with value `'before_validation'`)
or after (value `'after_validation'`) parsing and validation when they are [converted](dataclasses.md#stdlib-dataclasses-and-_pydantic_-dataclasses).

## Change behaviour globally

If you wish to change the behaviour of _pydantic_ globally, you can create your own custom `BaseModel`
Expand Down
7 changes: 4 additions & 3 deletions pydantic/__init__.py
Expand Up @@ -2,7 +2,7 @@
from . import dataclasses
from .annotated_types import create_model_from_namedtuple, create_model_from_typeddict
from .class_validators import root_validator, validator
from .config import BaseConfig, Extra
from .config import BaseConfig, ConfigDict, Extra
from .decorator import validate_arguments
from .env_settings import BaseSettings
from .error_wrappers import ValidationError
Expand All @@ -13,7 +13,7 @@
from .parse import Protocol
from .tools import *
from .types import *
from .version import VERSION
from .version import VERSION, compiled

__version__ = VERSION

Expand All @@ -30,6 +30,7 @@
'validator',
# config
'BaseConfig',
'ConfigDict',
'Extra',
# decorator
'validate_arguments',
Expand All @@ -42,7 +43,6 @@
'Required',
# main
'BaseModel',
'compiled',
'create_model',
'validate_model',
# network
Expand Down Expand Up @@ -120,5 +120,6 @@
'PastDate',
'FutureDate',
# version
'compiled',
'VERSION',
]
64 changes: 63 additions & 1 deletion pydantic/config.py
Expand Up @@ -4,6 +4,7 @@

from .typing import AnyCallable
from .utils import GetterDict
from .version import compiled

if TYPE_CHECKING:
from typing import overload
Expand All @@ -27,7 +28,7 @@ def __call__(self, schema: Dict[str, Any], model_class: Type[BaseModel]) -> None
else:
SchemaExtraCallable = Callable[..., None]

__all__ = 'BaseConfig', 'Extra', 'inherit_config', 'prepare_config'
__all__ = 'BaseConfig', 'ConfigDict', 'get_config', 'Extra', 'inherit_config', 'prepare_config'


class Extra(str, Enum):
Expand All @@ -36,6 +37,46 @@ class Extra(str, Enum):
forbid = 'forbid'


# https://github.com/cython/cython/issues/4003
# Will be fixed with Cython 3 but still in alpha right now
if not compiled:
from typing_extensions import Literal, TypedDict

class ConfigDict(TypedDict, total=False):
title: Optional[str]
anystr_lower: bool
anystr_strip_whitespace: bool
min_anystr_length: int
max_anystr_length: Optional[int]
validate_all: bool
extra: Extra
allow_mutation: bool
frozen: bool
allow_population_by_field_name: bool
use_enum_values: bool
fields: Dict[str, Union[str, Dict[str, str]]]
validate_assignment: bool
error_msg_templates: Dict[str, str]
arbitrary_types_allowed: bool
orm_mode: bool
getter_dict: Type[GetterDict]
alias_generator: Optional[Callable[[str], str]]
keep_untouched: Tuple[type, ...]
schema_extra: Union[Dict[str, Any], 'SchemaExtraCallable']
json_loads: Callable[[str], Any]
json_dumps: Callable[..., str]
json_encoders: Dict[Type[Any], AnyCallable]
underscore_attrs_are_private: bool

# whether or not inherited models as fields should be reconstructed as base model
copy_on_model_validation: bool
# whether dataclass `__post_init__` should be run after validation
post_init_call: Literal['before_validation', 'after_validation']

else:
ConfigDict = dict # type: ignore


class BaseConfig:
title: Optional[str] = None
anystr_lower: bool = False
Expand Down Expand Up @@ -66,6 +107,8 @@ class BaseConfig:
copy_on_model_validation: bool = True
# whether `Union` should check all allowed types before even trying to coerce
smart_union: bool = False
# whether dataclass `__post_init__` should be run before or after validation
post_init_call: Literal['before_validation', 'after_validation'] = 'before_validation'

@classmethod
def get_field_info(cls, name: str) -> Dict[str, Any]:
Expand Down Expand Up @@ -100,6 +143,25 @@ def prepare_field(cls, field: 'ModelField') -> None:
pass


def get_config(config: Union[ConfigDict, Type[BaseConfig], None]) -> Type[BaseConfig]:
if config is None:
return BaseConfig

else:
config_dict = (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if config is an child of BaseConfig this seems like a weird way to do things.Coupld we just return config in that case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing is most people write

class Config:
 ....

without actually setting BaseConfig as the parent and I felt like having a proper BaseConfig instance or child. I can remove this if you want or feel free to update

config
if isinstance(config, dict)
else {k: getattr(config, k) for k in dir(config) if not k.startswith('__')}
)

class Config(BaseConfig):
...

for k, v in config_dict.items():
setattr(Config, k, v)
return Config


def inherit_config(self_config: 'ConfigType', parent_config: 'ConfigType', **namespace: Any) -> 'ConfigType':
if not self_config:
base_classes: Tuple['ConfigType', ...] = (parent_config,)
Expand Down