Support Annotated type hints and extracting Field from Annotated #2147

JacobHayes · 2020-11-26T18:28:09Z

Change Summary

As discussed elsewhere (1, 2), Pydantic doesn't support typing.Annotated/typing_extensions.Annotated. When using an Annotated field, Pydantic raises:

TypeError: Fields of type "<class 'typing.Annotated'>" are not supported.

This PR adds support for Annotated by simply unwrapping the root type in ModelField. The full Annotated information is still visible with get_type_hints(x, include_extras=True) on the BaseModel subclasses.

I'm not sure if this needs doc updates, perhaps just a small note saying "Annotated type hints are supported"?

Separately, I'm working on a prototype for #2129 in https://github.com/JacobHayes/pydantic-annotated which expands on this functionality. I have a small check (based on the presence of pydantic.typing.Annotated) in there to do the unwrapping until this or similar is merged.

Related issue number

Decompose Field components into Annotated #2129

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

codecov · 2020-11-26T18:29:46Z

Codecov Report

Merging #2147 (3645dbf) into master (78934db) will decrease coverage by 0.02%.
The diff coverage is 98.33%.

@@             Coverage Diff             @@
##            master    #2147      +/-   ##
===========================================
- Coverage   100.00%   99.97%   -0.03%     
===========================================
  Files           23       23              
  Lines         4426     4477      +51     
  Branches       888      903      +15     
===========================================
+ Hits          4426     4476      +50     
- Misses           0        1       +1

Impacted Files	Coverage Δ
pydantic/typing.py	`99.41% <95.65%> (-0.59%)`	⬇️
pydantic/fields.py	`100.00% <100.00%> (ø)`
pydantic/schema.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78934db...3645dbf. Read the comment docs.

JacobHayes · 2020-11-27T05:27:55Z

~~This appears to be mostly working except when using a constrained type (gt, et al):~~

On field "x" the following field constraints are set but not enforced

~~Looks like I'll need a little more digging into the constraint logic to apply a get_origin or similar.~~

Tests are passing.

samuelcolvin · 2020-11-30T18:27:19Z

I think this looks like a great start, but I think before we merge we need:

Annotated[int, Field(alias='foobar')] to work - I guess we should raise an error if you set the default value by via the positional argument to Field
decent documentation on how to use Annotated, shouldn;t be that hard

JacobHayes · 2020-12-02T04:46:18Z

decent documentation on how to use Annotated, shouldn;t be that hard

I'll dig in. Planning to add two parts:

1 small comment on the Field Types page that they can be Annotated
a slightly larger comment on Schema saying Fields can be set in Annotated and some of the error cases

JacobHayes · 2020-12-02T05:19:59Z

@samuelcolvin any chance we could make typing_extensions a required dependency? Would certainly clean up some code/cases for Annotated, Literal, and maybe some other stuff in typing.py (but certainly not everything, particularly on 3.6 it seems).

JacobHayes · 2020-12-04T04:40:07Z

Looks like we need to either bump the docs builder to py3.9, install typing_extensions, or I can just inline the example (so it's not run). Any preferences?

I also added a few failing tests around defaults - I need to add:

error if the Field in Annotated has a default set
use the rhs default value when Field is in Annotated

JacobHayes · 2020-12-26T18:53:25Z

I don't think we can (cleanly) "error if the Field in Annotated has a default set" since we also need to support default_factory, unless:

we use default_factory from lhs Annotated[..., Field(...)] but default from rhs value (more complexity/error cases)
or, we inspect isinstance(rhs, Callable) to infer default_factory (larger/breaking change that should be separate if desirable, potentially a v2 change)

~~For now, I'll allow the defaults to be set in the lhs, but let me know if you prefer one of the two above.~~

edit: Error cases cleaned up a bit with some more refactoring, so I opted for case 1 above.

JacobHayes · 2020-12-26T23:50:34Z

@samuelcolvin I think this is ready for another look!

There may be a bit of a usage disconnect between default/default_factory that might be improved with this potential v2 change.

samuelcolvin

this is looking great, a few things to fix.

pydantic/fields.py

samuelcolvin · 2020-12-31T18:01:19Z

pydantic/typing.py

+    # typing_extensions.Annotated, so wrap them to short-circuit. We still want to use our wrapped
+    # get_origins defined above for non-Annotated data.
+
+    def get_args(tp: Type[Any], _get_args=get_args) -> Type[Any]:


I think get_args is already defined in this file, we should either:

use that version

add this logic to that method

give this a different name

1/3: We need the "Annotated aware" version in all places as the "default".

2: This probably gets tricky without larger "refactor now that typing-extensions is always available" changes, I think I'd have to add this logic to 3 versions of get_args and 2 versions of get_origin (the ones defined above) vs just wrapping whatever is decided above. Let me know if you'd like to just dup the check in those or refactor a bit (presuming #2147 (comment)).

I was able to refactor the get_args/get_origin stuff a bit with typing_extensions, but it's not a panacea:

those helper funcs aren't available on 3.6 so I kept the old "polyfills"

I had to remove two tests using a bare Callable (the typing_extensions functions seem to expect the Callable generics to be [...]). Might get just as messy again to add the shim if those are truly valid cases (creating a Model with a bare Callable field hint seemed to still work)

Looks like this makes the "test compiled without deps" fail (since we now import typing_extensions). Does this need to be tweaked somehow?

I'm content reverting those changes and/or doing something else if you prefer.

Given that ^ messiness, I've pulled those changes (will reserve them for a future typing_extensions PR). How do the latest commits look?

pydantic/typing.py

samuelcolvin · 2020-12-31T18:02:48Z

setup.py

@@ -130,7 +130,7 @@ def extra(self):
    ],
    extras_require={
        'email': ['email-validator>=1.0.3'],
-        'typing_extensions': ['typing-extensions>=3.7.2'],
+        'typing_extensions': ['typing-extensions>=3.7.4'],


I think we should move this to install_requires but perhaps that should be a separate PR.

I'm happy to do this in this PR - it ~~would~~ may greatly simplify #2147 (comment). Perhaps I leave a lot of the other cleanup out of this PR though to reduce noise.

I'll leave it out in the sake of getting this PR in. I have a separate branch with partial support I can push after this.

tests/test_main.py

JacobHayes · 2021-01-14T01:54:57Z

Good on my end - open discussions:

thomascobb · 2021-01-18T13:36:30Z

I've added a test for my use case in https://github.com/thomascobb/pydantic/blob/support-annotated-fields/tests/test_decorator.py:

def test_annotated_field_reuse():
    AOne = Annotated[int, Field(description='A long description of `one` we don\'t want to repeat')]

    class Model(BaseModel):
        one: AOne
        two: Annotated[int, Field(description='An optional integer')] = 3

        @classmethod
        @validate_arguments
        def alternative(
            cls,
            one: AOne,
            have_two: Annotated[bool, Field(description='Include two?')],
        ) -> 'Model':
            return cls(one=one, two=2 if have_two else 0)

    assert Model(one=1).dict() == dict(one=1, two=3)
    assert Model.alternative(one=1, have_two=False) == dict(one=1, two=0)

This is failing for a few reasons. First is in the type hint evaluations:

tests/test_decorator.py:134: in Model
    def alternative(
pydantic/decorator.py:61: in validate_arguments
    return validate(func)
pydantic/decorator.py:48: in validate
    vd = ValidatedFunction(_func, config)
pydantic/decorator.py:89: in __init__
    type_hints = get_type_hints(function)
/usr/lib/python3.8/typing.py:1264: in get_type_hints
    value = _eval_type(value, globalns, localns)
/usr/lib/python3.8/typing.py:270: in _eval_type
    return t._evaluate(globalns, localns)
/usr/lib/python3.8/typing.py:518: in _evaluate
    eval(self.__forward_code__, globalns, localns),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   NameError: name 'Model' is not defined

<string>:1: NameError

If I remove the -> 'Model' in the return type of alternative then I get:

tests/test_decorator.py:128: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pydantic/main.py:265: in __new__
    fields[ann_name] = inferred = ModelField.infer(
pydantic/fields.py:346: in infer
    field_info, value = cls._get_field_info(name, annotation, value, config)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

field_name = 'one'
annotation = typing_extensions.Annotated[int, FieldInfo(default=Ellipsis, description="A long description of `one` we don't want to repeat", extra={})]
value = PydanticUndefined, config = <class 'pydantic.main.Config'>

    @staticmethod
    def _get_field_info(
        field_name: str, annotation: Any, value: Any, config: Type['BaseConfig']
    ) -> Tuple[FieldInfo, Any]:
        """
        Get a FieldInfo from a root typing.Annotated annotation, value, or config default.
    
        The FieldInfo may be set in typing.Annotated or the value, but not both. If neither contain
        a FieldInfo, a new one will be created using the config.
    
        :param field_name: name of the field for use in error messages
        :param annotation: a type hint such as `str` or `Annotated[str, Field(..., min_length=5)]`
        :param value: the field's assigned value
        :param config: the model's config object
        :return: the FieldInfo contained in the `annotation`, the value, or a new one from the config.
        """
        field_info_from_config = config.get_field_info(field_name)
    
        field_info = None
        if get_origin(annotation) is Annotated:
            field_infos = [arg for arg in get_args(annotation)[1:] if isinstance(arg, FieldInfo)]
            if len(field_infos) > 1:
                raise ValueError(f'cannot specify multiple `Annotated` `Field`s for {field_name!r}')
            field_info = next(iter(field_infos), None)
            if field_info is not None:
                if field_info.default is not Undefined:
>                   raise ValueError(f'`Field` default cannot be set in `Annotated` for {field_name!r}')
E                   ValueError: `Field` default cannot be set in `Annotated` for 'one'

pydantic/fields.py:320: ValueError

Finally, if I run make lint I get:

tests/test_decorator.py:126:45: F722 syntax error in forward annotation "A long description of `one` we don't want to repeat"
tests/test_decorator.py:126:45: Q003 Change outer quotes to avoid escaping inner quotes
tests/test_decorator.py:130:47: F722 syntax error in forward annotation 'An optional integer'
tests/test_decorator.py:137:57: F722 syntax error in forward annotation 'Include two?'

Any ideas?

PrettyWood · 2021-01-18T16:21:49Z

@thomascobb The Q003 error is expected. Just change '...we don\'t...' into "...we don't..." (make format doesn't fix this automatically as it doesn't come from black but from flake8 🙄 )
All the other errors come from pyflakes. I already opened a PR and I just checked locally: it fixes all the rest of your issues.
Unfortunately this fix for pyflakes can't be merged yet because other issues have been spotted with it and I still haven't taken the time to dig into it

JacobHayes · 2021-01-18T18:59:44Z

@PrettyWood

Unfortunately this PR can't be merged because other issues have been spotted and I still haven't taken the time to dig into it

Thanks for looking into things! Do you mind linking to or describing some of the other issues (aside from @thomascobb's) in case I or others have a chance to check them out? If this gets too messy, I'm also happy to split this PR up into 2 parts - the first PR to just not break on unrelated Annotated uses (the first commit) and a second PR to support Field in Annotated (much messier 😁).

@thomascobb
I just pushed a commit that should fix that Field reuse issue - thanks for the find! I admittedly don't grok the distinction between Ellipsis and Undefined (anything aside from printing nicety?), but the FieldInfo mutation definitely adds a little complexity.

E NameError: name 'Model' is not defined

This part is unrelated to this PR/doesn't work on master unfortunately either.

broken minimum test_model_forward_ref

def test_model_forward_ref():
    from pydantic.decorator import validate_arguments

    class Model(BaseModel):
        one: int

        @classmethod
        @validate_arguments
        def alternative(cls, one) -> 'Model':  # still fails
            return cls(one=one)

I think it's due to the alternative method being created and validate_arguments called before Model is created. Perhaps there's a way to keep validate_arguments lazy here and do something in Model.update_forward_refs(), but in the short term you might have luck defining the function outside the class body.

working test_model_forward_ref

def test_model_forward_ref():
    from pydantic.decorator import validate_arguments

    class Model(BaseModel):
        one: int

    @classmethod
    @validate_arguments
    def alternative(cls, one) -> Model:
        return cls(one=one)

    Model.alternative = alternative

PrettyWood · 2021-01-18T20:12:21Z

The PR that can't be merged yet is the fix I made for pyflakes (I updated my message above to be more explicit. Sorry if this wasn't clear)

thomascobb · 2021-01-19T09:04:56Z

I just pushed a commit that should fix that Field reuse issue - thanks for the find! I admittedly don't grok the distinction between Ellipsis and Undefined (anything aside from printing nicety?), but the FieldInfo mutation definitely adds a little complexity.

Thanks, that fixes it

This part is unrelated to this PR/doesn't work on master unfortunately either.

Good point. The following works though, and it probably more correct...

from typing import TypeVar

def test_annotated_field_reuse():
    T = TypeVar("T")
    AOne = Annotated[int, Field(description='A long description of `one` we don\'t want to repeat')]

    class Model(BaseModel):
        one: AOne
        two: Annotated[int, Field(description='An optional integer')] = 3

        @classmethod
        @validate_arguments
        def alternative(
            cls: T,
            one: AOne,
            have_two: Annotated[bool, Field(description='Include two?')],
        ) -> T:
            return cls(one=one, two=2 if have_two else 0)

    assert Model(one=1).dict() == dict(one=1, two=3)
    assert Model.alternative(one=1, have_two=False) == dict(one=1, two=0)

samuelcolvin · 2021-02-13T14:41:26Z

I'm a bit unclear on this.

Is there anything (except conflicts which are trivial) which is blocking this? I would be keen to get it merged asap to include in v1.8 if possible.

JacobHayes · 2021-02-13T15:26:20Z

No blockers I'm aware of, just another round of reviews. I'll push a conflict fix for reqs in a min.

pydantic/fields.py

samuelcolvin · 2021-02-13T16:06:15Z

pydantic/typing.py

+        except ImportError:
+            # Create mock Annotated values distinct from `None`, which is a valid `get_origin`
+            # return value.
+            class _FalseMeta(type):


this is clever, even if it's sad we have to do it. 🙃

samuelcolvin · 2021-02-13T16:12:28Z

tests/test_annotated.py

+
+
+@pytest.mark.skipif(not Annotated, reason='typing_extensions not installed')
+@pytest.mark.parametrize(


I think this is okay, but as a piece of feedback, it's generally a good idea to include a simple test that demonstrates the basic behaviour at the top of test file. All this parametrize stuff is very clever but it's not that easy to read or reason with when things break.

You could also use pytest.importorskip here to avoid lots of skipif, or just pytestmark = ....

JacobHayes changed the title ~~Infer root type from Annotated (but toss away the rest of the annotation)~~ Infer root type from Annotated Nov 26, 2020

JacobHayes force-pushed the support-annotated-fields branch from dc9faa3 to b183faa Compare November 27, 2020 05:07

JacobHayes mentioned this pull request Nov 27, 2020

Simplified length limited string #2104

Closed

JacobHayes force-pushed the support-annotated-fields branch from 539abb1 to 3337876 Compare December 2, 2020 04:40

JacobHayes changed the title ~~Infer root type from Annotated~~ Support Annotated type hints and extracting Field from Annotated Dec 2, 2020

JacobHayes force-pushed the support-annotated-fields branch from 3337876 to b72dfa9 Compare December 2, 2020 04:44

JacobHayes force-pushed the support-annotated-fields branch from b72dfa9 to f5ba8e7 Compare December 2, 2020 04:50

JacobHayes force-pushed the support-annotated-fields branch 3 times, most recently from e3e5f48 to 6fd8826 Compare December 4, 2020 04:35

JacobHayes force-pushed the support-annotated-fields branch 5 times, most recently from 255621e to d7ac367 Compare December 26, 2020 23:42

JacobHayes force-pushed the support-annotated-fields branch from d7ac367 to 8e5d016 Compare December 26, 2020 23:56

samuelcolvin reviewed Jan 1, 2021

View reviewed changes

samuelcolvin mentioned this pull request Jan 1, 2021

Support Field(default_factory) in validate_arguments #2176

Merged

4 tasks

JacobHayes force-pushed the support-annotated-fields branch from 8e5d016 to 05b2ee5 Compare January 5, 2021 03:58

Infer root type from Annotated

687db0d

JacobHayes force-pushed the support-annotated-fields branch 2 times, most recently from fac50da to 769a006 Compare January 13, 2021 23:34

JacobHayes force-pushed the support-annotated-fields branch 3 times, most recently from bb6bd51 to 769a006 Compare January 14, 2021 00:03

JacobHayes added 2 commits January 13, 2021 18:04

Extract Field from Annotated

052fa08

Add changelog

aee676f

JacobHayes force-pushed the support-annotated-fields branch from 769a006 to aee676f Compare January 14, 2021 00:04

JacobHayes added 2 commits January 13, 2021 18:57

Extend existing get_args/get_origin

5bf718a

Fix Annotated on py3.6 without typing-extensions

9e40459

Handle Ellipsis default

1a7732e

JacobHayes mentioned this pull request Jan 15, 2021

Feature Request: Support for typing_extensions.Annotated (from PEP 593) #837

Closed

JacobHayes added 2 commits January 18, 2021 13:46

Fix field reuse after FieldInfo.default mutation

6835179

Fix ci

c3ea7f7

Merge branch 'master' into support-annotated-fields

3645dbf

samuelcolvin approved these changes Feb 13, 2021

View reviewed changes

samuelcolvin merged commit b742c6f into pydantic:master Feb 13, 2021

JacobHayes deleted the support-annotated-fields branch February 13, 2021 16:35

samuelcolvin mentioned this pull request Feb 17, 2021

fix coverage and make typing-extensions a required dependency #2368

Merged

PrettyWood mentioned this pull request Feb 23, 2021

[Feature Request] Provide a discriminated union type (OpenAPI 3) #619

Closed

This was referenced Mar 6, 2021

Bump pydantic from 1.7.3 to 1.8.1 in /backend Vaarna/vaarna#139

Merged

Bump pydantic from 1.6.1 to 1.8.1 in /emmet-core rkingsbury/emmet#25

Closed

dependabot bot mentioned this pull request Mar 17, 2021

Bump pydantic[dotenv] from 1.7.3 to 1.8.1 enchant97/web-portal#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Annotated type hints and extracting Field from Annotated #2147

Support Annotated type hints and extracting Field from Annotated #2147

JacobHayes commented Nov 26, 2020 •

edited

codecov bot commented Nov 26, 2020 •

edited

JacobHayes commented Nov 27, 2020 •

edited

samuelcolvin commented Nov 30, 2020

JacobHayes commented Dec 2, 2020 •

edited

JacobHayes commented Dec 2, 2020 •

edited

JacobHayes commented Dec 4, 2020

JacobHayes commented Dec 26, 2020 •

edited

JacobHayes commented Dec 26, 2020

samuelcolvin left a comment

samuelcolvin Dec 31, 2020

JacobHayes Jan 5, 2021 •

edited

JacobHayes Jan 5, 2021 •

edited

JacobHayes Jan 14, 2021 •

edited

samuelcolvin Dec 31, 2020

JacobHayes Jan 5, 2021 •

edited

JacobHayes Jan 13, 2021

JacobHayes commented Jan 14, 2021

thomascobb commented Jan 18, 2021

PrettyWood commented Jan 18, 2021 •

edited

JacobHayes commented Jan 18, 2021 •

edited

PrettyWood commented Jan 18, 2021 •

edited

thomascobb commented Jan 19, 2021

samuelcolvin commented Feb 13, 2021

JacobHayes commented Feb 13, 2021

samuelcolvin Feb 13, 2021

samuelcolvin Feb 13, 2021



		@pytest.mark.skipif(not Annotated, reason='typing_extensions not installed')
		@pytest.mark.parametrize(

Support Annotated type hints and extracting Field from Annotated #2147

Support Annotated type hints and extracting Field from Annotated #2147

Conversation

JacobHayes commented Nov 26, 2020 • edited

Change Summary

Related issue number

Checklist

codecov bot commented Nov 26, 2020 • edited

Codecov Report

JacobHayes commented Nov 27, 2020 • edited

samuelcolvin commented Nov 30, 2020

JacobHayes commented Dec 2, 2020 • edited

JacobHayes commented Dec 2, 2020 • edited

JacobHayes commented Dec 4, 2020

JacobHayes commented Dec 26, 2020 • edited

JacobHayes commented Dec 26, 2020

samuelcolvin left a comment

Choose a reason for hiding this comment

samuelcolvin Dec 31, 2020

Choose a reason for hiding this comment

JacobHayes Jan 5, 2021 • edited

Choose a reason for hiding this comment

JacobHayes Jan 5, 2021 • edited

Choose a reason for hiding this comment

JacobHayes Jan 14, 2021 • edited

Choose a reason for hiding this comment

samuelcolvin Dec 31, 2020

Choose a reason for hiding this comment

JacobHayes Jan 5, 2021 • edited

Choose a reason for hiding this comment

JacobHayes Jan 13, 2021

Choose a reason for hiding this comment

JacobHayes commented Jan 14, 2021

thomascobb commented Jan 18, 2021

PrettyWood commented Jan 18, 2021 • edited

JacobHayes commented Jan 18, 2021 • edited

PrettyWood commented Jan 18, 2021 • edited

thomascobb commented Jan 19, 2021

samuelcolvin commented Feb 13, 2021

JacobHayes commented Feb 13, 2021

samuelcolvin Feb 13, 2021

Choose a reason for hiding this comment

samuelcolvin Feb 13, 2021

Choose a reason for hiding this comment

JacobHayes commented Nov 26, 2020 •

edited

codecov bot commented Nov 26, 2020 •

edited

JacobHayes commented Nov 27, 2020 •

edited

JacobHayes commented Dec 2, 2020 •

edited

JacobHayes commented Dec 2, 2020 •

edited

JacobHayes commented Dec 26, 2020 •

edited

JacobHayes Jan 5, 2021 •

edited

JacobHayes Jan 5, 2021 •

edited

JacobHayes Jan 14, 2021 •

edited

JacobHayes Jan 5, 2021 •

edited

PrettyWood commented Jan 18, 2021 •

edited

JacobHayes commented Jan 18, 2021 •

edited

PrettyWood commented Jan 18, 2021 •

edited