feat: add computed fields #2625

PrettyWood · 2021-04-02T22:47:27Z

Change Summary

I'm waiting for feedback on the API and doc 🙏

`property`-like syntax

from functools import cached_property
from random import randint

from pydantic import BaseModel, computed_field


class Square(BaseModel, underscore_attrs_are_private=True):
    side: float
    _private_attr: int = 1

    @computed_field(title='The area')
    @property
    def area(self) -> float:
        return self.side ** 2

    @area.setter
    def area(self, area: float) -> None:
        self.side = area ** .5

    @area.deleter
    def area(self) -> None:
        self.side = 0

    @computed_field(alias='The random number')
    @cached_property
    def random_n(self) -> int:
        return randint(0, 1_000)

    @property
    def public_attr(self) -> int:
        return self._private_attr + 1

    @public_attr.setter
    def public_attr(self, v: int) -> None:
        self._private_attr = v - 1


sq = Square(side=10)
assert sq._private_attr == 1
assert sq.public_attr == 2
sq.public_attr = 5
assert sq._private_attr == 4
assert sq.public_attr == 5
the_random_n = sq.random_n
assert sq.dict(by_alias=True) == {'side': 10, 'area': 100, 'The random number': the_random_n}
sq.area = 49
assert sq.dict() == {'side': 7, 'area': 49, 'random_n': the_random_n}
del sq.area
assert sq.dict() == {'side': 0, 'area': 0, 'random_n': the_random_n}
assert sq.dict(exclude={'random_n'}) == {'side': 0, 'area': 0}

`ComputedField` syntax

from pydantic import BaseModel, ComputedField, ValidationError, validator


class User(BaseModel, validate_assignment=True):
    first_name: str
    surname: str
    full_name: str = ComputedField(lambda self: f'{self.first_name} {self.surname}')

    @validator('full_name')
    def should_not_have_pika(cls, v: str) -> str:
        if 'pika' in v.lower():
            raise ValueError('We love pika but not that much!')
        return v


u = User(first_name='John', surname='Smith')
assert u.dict() == {'first_name': 'John', 'surname': 'Smith', 'full_name': 'John Smith'}
u.first_name = 'Mike'
assert u.dict()['full_name'] == 'Mike Smith'

try:
    User(first_name='Pika', surname='Chu')
    assert False
except ValidationError:
    # pydantic.error_wrappers.ValidationError: 1 validation error for User
    # full_name
    #   We love pika but not that much! (type=value_error)
    assert True

See the tests for more examples

Related issue number

closes #935
closes #1241
closes #2313

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

PrettyWood · 2021-04-05T08:45:19Z

Writing only @computed_field instead of @field with @property is annoying because of mypy and IDE. Writing

If TYPE_CHECKING:
    computed_field = property

won't work with kwargs like alias and stuff.
And it won't work fine with property-like descriptors

This is why I chose to keep the combo @field + @property, which is explicit, just a few characters longer and well supported without any effort. The only issue is that decorating property itself is not yet handled by mypy but it's a known issue that may be solved in the near future.

If we want to change the API and need to work on a better mypy support fine but I won't do it myself sorry: I have not the will nor the energy to work on the mypy plugin since it's a lot of work for IMHO not a big gain.

yicone · 2022-05-31T07:01:27Z

When reference a property with @comuted_field annotation, the type hint of the reference is gone.

I'm not sure if the bug comes from this branch or the Python extension of VS Code currently.

E.g.:

@property
def A(self) -> int:
	print(self.B)  # actual is "(method) B: Any", expect is "(property) B: List[float]"
	return 1

@computed_field
@property
def B(self) -> List[float]:
	return []

Editor version info:

Visual Stuido Code: 1.67.2
with Python extension : v2022.6.2

Pydantic version info:

         pydantic version: 1.9.0
        pydantic compiled: False
             install path: /Users/Tr/Workspace/yicone/pydantic/pydantic
           python version: 3.9.12 (main, Mar 26 2022, 15:51:15)  [Clang 13.1.6 (clang-1316.0.21.2)]
                 platform: macOS-12.2-x86_64-i386-64bit
 optional deps. installed: ['typing-extensions']

david-saeger · 2022-06-08T20:56:35Z

any reason there hasnt been movement on this in a while? are we ever going to get it?

Bobronium · 2022-06-08T21:15:46Z

@yicone, can reproduce. Also can fix by replacing Any to ~T in annotation for func argument and return annotation of computed_field.

Bobronium

Fixes issue described in #2625 (comment)

Tested on VSCode, please also confirm that there's no regression in PyCharm.

Bobronium · 2022-06-08T21:16:57Z

pydantic/fields.py

+
+@overload
+def computed_field(
+    func: Any,


Suggested change

func: Any,

func: T,

Bobronium · 2022-06-08T21:17:18Z

pydantic/fields.py

+    exclude: Optional['ExcludeInclude'] = None,
+    include: Optional['ExcludeInclude'] = None,
+    repr: bool = True,
+) -> Any:


Suggested change

) -> Any:

) -> T:

Bobronium · 2022-06-08T21:17:22Z

pydantic/fields.py

+
+
+def computed_field(
+    func: Optional[Any] = None,


Suggested change

func: Optional[Any] = None,

func: Optional[T] = None,

Bobronium · 2022-06-08T21:17:33Z

pydantic/fields.py

+    exclude: Optional['ExcludeInclude'] = None,
+    include: Optional['ExcludeInclude'] = None,
+    repr: bool = True,
+) -> 'Union[Callable[[Any], ComputedField], ComputedField]':


Suggested change

) -> 'Union[Callable[[Any], ComputedField], ComputedField]':

) -> 'Union[Callable[[T], ComputedField], ComputedField]':

Bobronium · 2022-06-08T22:03:21Z

@pawelswiecki I'd suggest not to dig down into the rabbit hole when it comes to the features that can be implemented after the release. I think changes should be gradual and easy to make. It's possible to add support for @abstractmethod after releasing this. Removing such support is not as trivial though, since it would introduce breaking changes.

Let's focus on things that can't be easily changed/fixed after release.

As for your example, let me suggest this (click here):

from enum import Enum
from typing_extensions import Self

from pydantic import BaseModel, Field

class EventType(Enum):
    JOB_STARTED =  "job_started"
    JOB_FINISHED =  "job_finished"

class BaseEvent(BaseModel):
    id: str
    event_type: EventType

    def __init_subclass__(cls, **kwargs) -> None:
        return super().__init_subclass__(**kwargs)
        if (event_type_field := cls.__fields__.get('event_type')) is None or not event_type_field.field_info.const:
            raise NotImplementedError('You must define event_type for concrete event')

    @classmethod
    def create(cls, **data) -> Self:
        id_ = "generated id"
        return cls(id=id_, **data)

class JobStartedEvent(BaseEvent):
    event_type = Field(EventType.JOB_STARTED, const=True)

class JobFinishedEvent(BaseEvent):
    event_type = Field(EventType.JOB_FINISHED, const=True)


print(JobFinishedEvent.create().event_type)  # EventType.JOB_FINISHED


class InvalidEvent(BaseEvent):  ...  # NotImplementedError: You must define event_type for concrete event

Bobronium · 2022-06-08T22:29:18Z

To summarize, things to do before merging:

Fix merge conflicts
Fix typing: feat: add computed fields #2625 (review)
Fix inheritance: feat: add computed fields #2625 (comment)
- raise AttributeError in ComputedField.__getattr__ when condition is not met
- add tests that include inheritance
Decide whether to support @abstractmethod (IMO, no, see feat: add computed fields #2625 (comment))

Is there something else I'm missing?

sorry for so many comments

PrettyWood · 2022-06-08T23:37:42Z

@Bobronium thanks for the recap. I'll work on it this weekend. Sorry for being inactive, just a lot on my plate with both professional and personal matters

Bobronium · 2022-06-09T00:33:48Z

@prettywoo, can relate. Please take care!

Also, feel free to ask for any help if needed.

caniko · 2022-06-10T09:42:25Z

I am so excited for this feature to be implemented.

Is there something else I'm missing?

Yes @Bobronium, things to consider:

Testing with frozen=true
Validation feature, and testing
Testing that combines classmethod with computed_property:
- Combined with property
- Combined with cached_property
Method for and testing flushing cache of cached_property based computed_fields of both object and class

Thank you all so much for your hard work, it really does make my life much better at the very least. I can also help!

caniko · 2022-07-29T14:52:45Z

Suggestion:

I don't disagree with this, and think it makes most sense:

...
    @computed_field
    @cached_property
    def hard_compute():
        <very hard problem>
...

However, I suggest that we also introduce shorthand notation for property and cached_property:

    @cached_computed_field_property
    ...

And:

    @computed_field_property
    ...

This is nice when computed_fields are used frequently in a package, like my case. I don't want to end up in decorator-hell. Hope this makes sense.

caniko · 2022-07-29T14:55:52Z

Feature suggestion:
Can we make computed_fields work with numba.jit decorator where property and cached_property is optional?

I realize that this is out of scope for this feature, but perhaps use the underlying backend for jit? Like computed -> computed_field. Maybe this would fall under computed.

@computed(field_a, field_b, ...)
@jit
def some_func(field_a, field_b, ...):
    ...

bjmc · 2022-07-29T14:57:06Z

cached_computed_field_property is rather long.

What about something like?

@computed_field(cache=True)
def calculate(self):
    pass

caniko · 2022-07-29T15:02:58Z

@bjmc, that is also an option. Where property is the default; however, it would interfere with the default behavior:

...
    @computed_field
    @property
    def hard_compute():
        <very hard problem>
...

Hence, my original suggestion.

samuelcolvin · 2022-08-04T11:13:54Z

This won't be added for v1.10. I don't know if we can use some of this logic in V2 or whether we'll need a new implementation.

pawelswiecki · 2022-08-04T11:31:25Z

@Bobronium Thank you for your suggestion, with the working code. Looking good, much appreciated!

PrettyWood · 2022-08-10T22:01:43Z

For those waiting for this feature, it will be 100% added in v2!
Even though we may change the implementation, we got a lot of precious feedbacks (thanks everyone ❤️) on the desired APIs: a "property-like" one and a more "type annotation" one.
So we will most likely have these examples work the same 👍

EDIT: I'll make sure #2625 (comment) and #2625 (comment) are checked 😉

Bobronium · 2022-08-11T00:20:15Z

Wow. Looks like pylance (pyright) supports custom defined properties through simple descriptor typing.

I've managed to create a computed_field type that is interpreted by VSCode as a property (borrowed code for generic property here: python/typing#985 (comment)):

from __future__ import annotations
from random import randint
from typing import Any, Callable, Generic, NoReturn, TypeVar, overload



_G1 = TypeVar("_G1")
_G2 = TypeVar("_G2")
_S1 = TypeVar("_S1")
_S2 = TypeVar("_S2")


class computed_field(Generic[_G1, _S1]):
    fget: Callable[[Any], _G1] | None
    fset: Callable[[Any, _S1], None] | None
    fdel: Callable[[Any], None] | None

    @overload
    def __new__(
        cls,
        fget: Callable[[Any], _G2],
        fset: None = ...,
        fdel: Callable[[Any], None] | None = ...,
        doc: str | None = ...,
    ) -> computed_field[_G2, NoReturn]:
        ...

    @overload
    def __new__(
        cls,
        fget: Callable[[Any], _G2],
        fset: Callable[[Any, _S2], None],
        fdel: Callable[[Any], None] | None = ...,
        doc: str | None = ...,
    ) -> computed_field[_G2, _S2]:
        ...

    @overload
    def __new__(
        cls,
        fget: None = ...,
        fset: None = ...,
        fdel: Callable[[Any], None] | None = ...,
        alias: str | None = ...,
        title: str | None = ...,
        description: str | None = ...,
        exclude: set[str] | None = ...,
        include: set[str] | None = ...,
        repr: bool = True,
        doc: str | None = ...,
    ) -> computed_field[NoReturn, NoReturn]:
        ...

    def __call__(self: computed_field[NoReturn, NoReturn], fget: Callable[[Any], _G2]) -> computed_field[_G2, NoReturn]:
        ...

    def getter(self, __fget: Callable[[Any], _G2]) -> computed_field[_G2, _S1]:
        ...

    def setter(self, __fset: Callable[[Any, _S2], None]) -> computed_field[_G1, _S2]:
        ...

    def deleter(self, __fdel: Callable[[Any], None]) -> computed_field[_G1, _S1]:
        ...

    def __get__(self, __obj: Any, __type: type | None = ...) -> _G1:
        ...

    def __set__(self, __obj: Any, __value: _S1) -> None:
        ...

    def __delete__(self, __obj: Any) -> None:
        ...


class Square:
    side: float

    @computed_field
    def area(self) -> float:
        return self.side**2

    @computed_field(alias="The random number")
    def random_n(self) -> int:
        return randint(0, 1_000)


reveal_type(Square().area)
reveal_type(Square().random_n)

Type of "Square().area" is "float"
Type of "Square().random_n" is "int"

Unfurtunately, both ~~mypy~~ and pycharm still don't know how to handle (generic) descriptors.

Actually, mypy also is able to infer properties types here!

caniko · 2022-09-01T13:43:35Z

Another thing that I thought of is to include a keyword argument for computing all computed fields when running model.dict() or model.json().

Example:

model.dict(call_computed_fields=True)

Will evaluate and provide dict with all computed fields computed.

nickswiss · 2023-01-12T20:02:20Z

@PrettyWood Any update on this feature, think it is a great addition and gets rid of accomplishing this in validators. Thank you!

* implement computed fields 🎉 * dataclass properties * catch errors in properties * fix include and exclude * add example from pydantic/pydantic#2625, fix by_alias * fix on older python

samuelcolvin · 2023-04-18T11:44:29Z

Thank you all (especially @PrettyWood) for 2 years (! 🙈) of patience.

I've migrated this feature to pydantic V2 and pydantic-core, see #5502.

Thank you so much @PrettyWood for your work on this, I've used some of your code, and most of your tests and docs in the new PR 🙇 🙏.

PrettyWood force-pushed the field-property branch 2 times, most recently from b4ed70b to 11d6ac9 Compare April 2, 2021 22:50

PrettyWood mentioned this pull request Apr 2, 2021

Serializing @property #935

Closed

PrettyWood force-pushed the field-property branch from 11d6ac9 to 9d8bd6e Compare April 2, 2021 23:33

PrettyWood changed the title ~~feat: add computed fields (only getters for now)~~ feat: add computed fields Apr 2, 2021

PrettyWood force-pushed the field-property branch 7 times, most recently from 80363ea to 54e23c5 Compare April 3, 2021 14:28

PrettyWood marked this pull request as ready for review April 3, 2021 14:28

PrettyWood force-pushed the field-property branch 2 times, most recently from 7d8521c to 83a3916 Compare April 5, 2021 08:41

PrettyWood added 12 commits April 6, 2021 21:38

feat: add computed fields (only getters for now)

000cb7f

feat: add setter support

55535c0

feat: handle __del__ and cached_property

ee2c391

refactor: add readOnly only if no setter

ae70764

fix: coverage

d4c099f

simplify a bit

21863f2

😏

ce12ab0

add tests for single dispatch

bbfc4d2

fix: avoid mypy ignore

eac0f39

basic mypy support

55ddb1a

fix: serialize properly and add exclude_computed

4097d7d

chore: add basic change file

797be54

PrettyWood force-pushed the field-property branch 2 times, most recently from 9926afb to 3f775d1 Compare April 6, 2021 19:40

Bobronium suggested changes Jun 8, 2022

View reviewed changes

samuelcolvin added deferred deferred until future release or until something else gets done and removed awaiting author revision labels Aug 4, 2022

samuelcolvin mentioned this pull request Oct 11, 2022

Mistakenly including cached_property as a field #4609

Closed

15 tasks

caniko mentioned this pull request Oct 24, 2022

Overview of computed fields 🧮 #4650

Closed

11 tasks

ironkayman mentioned this pull request Nov 30, 2022

Item better config extraction and parsing: move to PyDantic classes ironkayman/tftm-alien-invasion#14

Closed

lsetiawan mentioned this pull request Mar 2, 2023

Setup initial configuration file spec for solver seafloor-geodesy/gnatss#57

Merged

samuelcolvin added a commit to pydantic/pydantic-core that referenced this pull request Apr 16, 2023

add example from pydantic/pydantic#2625, fix by_alias

0b1859c

samuelcolvin mentioned this pull request Apr 17, 2023

Computed Fields #5502

Merged

1 task

samuelcolvin closed this Apr 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add computed fields #2625

feat: add computed fields #2625

PrettyWood commented Apr 2, 2021 •

edited

PrettyWood commented Apr 5, 2021

yicone commented May 31, 2022

david-saeger commented Jun 8, 2022

Bobronium commented Jun 8, 2022 •

edited

Bobronium left a comment •

edited

Bobronium Jun 8, 2022

Bobronium Jun 8, 2022

Bobronium Jun 8, 2022

Bobronium Jun 8, 2022

Bobronium commented Jun 8, 2022 •

edited

Bobronium commented Jun 8, 2022 •

edited

PrettyWood commented Jun 8, 2022

Bobronium commented Jun 9, 2022 •

edited

caniko commented Jun 10, 2022 •

edited

caniko commented Jul 29, 2022 •

edited

caniko commented Jul 29, 2022 •

edited

bjmc commented Jul 29, 2022

caniko commented Jul 29, 2022 •

edited

samuelcolvin commented Aug 4, 2022

pawelswiecki commented Aug 4, 2022 •

edited

PrettyWood commented Aug 10, 2022 •

edited

Bobronium commented Aug 11, 2022 •

edited

caniko commented Sep 1, 2022 •

edited

nickswiss commented Jan 12, 2023

samuelcolvin commented Apr 18, 2023

	) -> 'Union[Callable[[Any], ComputedField], ComputedField]':
	) -> 'Union[Callable[[T], ComputedField], ComputedField]':

feat: add computed fields #2625

feat: add computed fields #2625

Conversation

PrettyWood commented Apr 2, 2021 • edited

Change Summary

property-like syntax

ComputedField syntax

Related issue number

Checklist

PrettyWood commented Apr 5, 2021

yicone commented May 31, 2022

david-saeger commented Jun 8, 2022

Bobronium commented Jun 8, 2022 • edited

Bobronium left a comment • edited

Choose a reason for hiding this comment

Bobronium Jun 8, 2022

Choose a reason for hiding this comment

Bobronium Jun 8, 2022

Choose a reason for hiding this comment

Bobronium Jun 8, 2022

Choose a reason for hiding this comment

Bobronium Jun 8, 2022

Choose a reason for hiding this comment

Bobronium commented Jun 8, 2022 • edited

Bobronium commented Jun 8, 2022 • edited

sorry for so many comments

PrettyWood commented Jun 8, 2022

Bobronium commented Jun 9, 2022 • edited

caniko commented Jun 10, 2022 • edited

caniko commented Jul 29, 2022 • edited

caniko commented Jul 29, 2022 • edited

bjmc commented Jul 29, 2022

caniko commented Jul 29, 2022 • edited

samuelcolvin commented Aug 4, 2022

pawelswiecki commented Aug 4, 2022 • edited

PrettyWood commented Aug 10, 2022 • edited

Bobronium commented Aug 11, 2022 • edited

caniko commented Sep 1, 2022 • edited

nickswiss commented Jan 12, 2023

samuelcolvin commented Apr 18, 2023

PrettyWood commented Apr 2, 2021 •

edited

`property`-like syntax

`ComputedField` syntax

Bobronium commented Jun 8, 2022 •

edited

Bobronium left a comment •

edited

Bobronium commented Jun 8, 2022 •

edited

Bobronium commented Jun 8, 2022 •

edited

Bobronium commented Jun 9, 2022 •

edited

caniko commented Jun 10, 2022 •

edited

caniko commented Jul 29, 2022 •

edited

caniko commented Jul 29, 2022 •

edited

caniko commented Jul 29, 2022 •

edited

pawelswiecki commented Aug 4, 2022 •

edited

PrettyWood commented Aug 10, 2022 •

edited

Bobronium commented Aug 11, 2022 •

edited

caniko commented Sep 1, 2022 •

edited