validate_assignment decorator improvements

[samuelcolvin]

Future improvements to the validate_assignment decorator #1179:

• arguments to the decorator, including: validators, custom config (partially fixed by Valdiate arguments config #1663), return value validation
• rewrite front page to explain the 3 or 4 primary interfaces to pydantic:
  • BaseModel
  • dataclasses
  • parse_obj_as
  • validate_arguments
• perhaps change the error raised for invalid arguments to inherit from TypeError as other argument errors do. (Currently the normal ValidationError is raised which inherits from ValueError)
• allow validate_assignment to be set in "strict" mode so types are not coerced, this is really just Strict configuration #1098
• option to enable or disable validation via an environment variable or similar - e.g. to only use the function in development or testing or on a subset of calls
• option to still call the function if validation fails
• get rid of the extra arguments to the underlying model so it's useful
• instance methods @validate_arguments on instance methods #1222

[RileyMShea]

Python Version: 3.8
Pydantic Version: 1.17
Vscode: v1.50.1
Pylance: v2020.10.2

Using the decorator on v1.17 with arguments seems to mangle the function signature in vscode, specifically with their pylance language server. Haven't tested jedi or microsoft language servers.

@validate_arguments(config=dict(arbitrary_types_allowed=True))
# or
@validate_arguments(config=None)

# modifies signature from def foo(a:int,b:str)->str:... to def foo()->Any:

The validation still seems to work correctly, but all intellisense is lost.

---

@validate_arguments by itself works as expected though

Pycharm with pydantic extension does show correct signature on hover though.

[Kilo59]

The only feature so far that I wish @validate_arguments could provide is to use_enum_value.

I ran into this in the context of building a convenience interface to a web service that takes some query parameters.

Example
https://petstore.swagger.io/#/pet/findPetsByStatus

import enum
from typing import List

import httpx
from pydantic import validate_arguments

class Status(str, enum.Enum):
    available = "available"
    pending = "pending"
    sold = "sold"

@validate_arguments
def find_pets_by_status(status: List[Status]) -> List[dict]:
    r = httpx.get("https://petstore.swagger.io/v2/pet/findByStatus", params={"status": status})
    print(r.request.url)
    return r.json()

print(find_pets_by_status(["pending", "sold"]))
>>>https://petstore.swagger.io/v2/pet/findByStatus?status=Status.sold&status=Status.pending
>>>[]

The solution is easy enough, just extract the values for the enum.
[s.value for s in status]
Nonetheless, I miss use_enum_value 😄 .

Is this perhaps something we could just make default behavior if the enum is a mixin type?

[markedwards]

I think validate_arguments should support an underscore arg, for args that are ignored in the body. Right now it fails with:

 File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Foo
_
  extra fields not permitted (type=value_error.extra)

This is not to say that such a field should be validated, it should simply be ignored by validate_arguments, because it is an argument that is only there because it must be in the signature and is not otherwise used.

[markedwards]

My use case for validate_arguments is to gather any errors related to the argument validation, and return them after applying custom formatting. So I do something like:

try:
    result = validate_arguments(config=dict(arbitrary_types_allowed=True))(fn)(*args, **kwargs)
except PydanticValidationError as e:
    # apply formatting to errors

The issue with this is it also catches any PydanticValidationError exceptions that occur in the body of function, and I only want to catch and format the errors related to the input.

It would be useful to be able to get the validated arguments separately from executing the function. Is this possible already, without effectively writing my own version of validate_arguments?

[PrettyWood]

Hi @markedwards
Have you seen https://pydantic-docs.helpmanual.io/usage/validation_decorator/#validate-without-calling-the-function?

[markedwards]

@PrettyWood I had not, and that seems to do the job. Thanks! One minor nitpick I would mention is mypy can't understand it, but that isn't a showstopper.

What I ideally want is a way to get the validated arguments (handling exceptions), and then call the function with the validated arguments. Is there a way to get the validated args and kwargs from validate(), so I can avoid re-validating?

[rsokl]

Note that #1272 added support for instance-methods; the checkbox

should be checked.

[RobertCraigie]

Not sure if this is the correct place for this issue but I've been trying to add the @validate_arguments function to https://github.com/RobertCraigie/prisma-client-py and it gets stuck in an infinite loop when the function is called.

Pydantic seems to continuously call create_model for the TypedDict types I'm using (there are a lot of auto-generated and nested types). Naively patching create_model_from_typeddict to cache models fixes the issue.

from pydantic import annotated_types

create_model_from_typeddict = annotated_types.create_model_from_typeddict

def patched_create_model(
    typeddict_cls: Type[Any], **kwargs: Any
) -> Type[BaseModel]:
    if hasattr(typeddict_cls, '__pydantic_model__'):
        return typeddict_cls.__pydantic_model__

    kwargs.setdefault('__module__', typeddict_cls.__module__)
    model = create_model_from_typeddict(typeddict_cls, **kwargs)
    typeddict_cls.__pydantic_model__ = model
    return model

annotated_types.create_model_from_typeddict = patched_create_model

I'm using the development version of pydantic, installed from GitHub.

Here's a minimal repro that will crash with a recursion error:

from pydantic import validate_arguments
from typing import Optional, TypedDict


class UserCreateInput(TypedDict):
    name: str
    post: 'PostCreateInput'


class PostCreateInput(TypedDict):
    title: str
    author: Optional['UserCreateInput']


@validate_arguments
def create_user(data: UserCreateInput) -> None:
    print(data)


create_user({'name': 'Robert'})

Edit: Just realised an issue was just created for this bug: #3297

[RobertCraigie]

I would also suggest adding an argument to make model creation lazy.

I'm using validate_arguments on a lot of auto-generated functions with a lot of recursive / nested types and this significantly increases import time and memory usage.

e.g. @validate_arguments(lazy=True)

[markedwards]

Just following up on my earlier question above about getting the validated args and kwargs, I see that this is possible:

def validate(fn):
    validator = validate_arguments()(fn)

    @wraps(fn)
    def wrapped(*args, **kwargs):
        validated = validator.validate(*args, **kwargs)
        validated_args = [getattr(validated, validator.vd.arg_mapping[I]) for i in range(len(args))]
        validated_kwargs = {k: getattr(validated, k) for k in kwargs.keys()}

Can someone speak to whether there is a better way to achieve this? Better as in less hacky and likely to continue to work?

[PhillSimonds]

Hi all!

I wrote a decorator to do custom validation on fields and args when a method is called. Pydantic does custom validation of fields at object instantiation, which works for many use cases but is inflexible for others (e.g. for optional fields that are used only in some methods). Pydantic can do basic validation on arguments, but not custom validation. Please correct me if I'm wrong on either of those two points :). The logic looks something like this:

    @run_custom_validations(
        [
            FieldValidator(validator=associated_device_has_primary_ip, field_names=("intf_a", "intf_z")),
            FieldValidator(validator=intf_has_one_ip_address, field_names=("intf_a", "intf_z")),
            ArgumentValidator(validator=integer_is_prime, arg_strings=("count",)),
        ]
    )
    def ping(self, count: int = 5) -> dict:

The validator object takes a callable and the string representation of either the fields or the args. The value of each field/arg is passed into the validator function, which raises a ValueError in the case that it says the data is invalid. The decorator logic catches all ValueErrors, then raises an error at the end with all of the ValueErrors included.

I'm curious if anyone else has a need for something like this, and if a contribution back makes sense? I also don't know if here is the right place to post on it, so let me know if I should redirect elsewhere :).

[m-alisafaee]

Hi all,

The default value for the arbitrary_types_allowed parameter of validate_arguments should be set to True (currently it defaults to False). User types are passed to functions in many cases (about 90% in my code) and it's a bit inconvenient to set it every time. Moreover, a default value of True shouldn't cause any harm.

[ErnestoLoma]

Hi all,
I have a class method that is used to create an instance of the class. This method has the validate_arguments annotation. Everything is fine until I add a type annotation for the return type. Then I get a NameError name is not defined. Any suggestions would be appreciated. Below is a snippet that demonstrates the problem:
``from future import annotations

from typing import Optional

from pydantic import BaseModel, validate_arguments

class User(BaseModel):
class Config:
validate_assignment = True

id: int
name: str = "Jane Doe"
friend: Optional[User] = None

def add_friend(self, friend: User):
    self.friend = friend

@classmethod
@validate_arguments()
def create_bob(cls, id: int) -> User:
    return cls(id=id, name="Bob")

User.update_forward_refs()
if name == "main":
bob = User.create_bob(1)

[jayaddison]

Re: @validate_arguments

I'm not exactly sure how to phrase this feedback yet, but would like to share some thoughts. Perhaps this minimal-ish repro case is a start:

Python 3.11.2 (main, Feb 12 2023, 00:48:52) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydantic
>>> @pydantic.validate_arguments
... def foo(a: float | int):
...     print(a)
... 
>>> foo(1)
1.0

[the-matt-morris]

Re: @validate_arguments

I'm not exactly sure how to phrase this feedback yet, but would like to share some thoughts. Perhaps this minimal-ish repro case is a start:

Python 3.11.2 (main, Feb 12 2023, 00:48:52) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydantic
>>> @pydantic.validate_arguments
... def foo(a: float | int):
...     print(a)
... 
>>> foo(1)
1.0

@jayaddison I think you're looking for the smart_union config option here:

import pydantic

@pydantic.validate_arguments(config={"smart_union": True})
def foo(a: float | int):
    print(a)

foo(1)
1

I've run into the same thing and stumbled upon smart_union at some point. It does have some caveats, but works well in this use case.

[jayaddison]

Perfect - thanks, @the-matt-morris - adding smart_union produces the behaviour that I was expecting.

And now I've read that there's a possibility that smart_union may be enabled by default after strict mode becomes available - so I think that resolves my feedback here. Thanks again.

[jnj16180340]

Is validate_arguments intended to work with dataclass and classmethod? I realize the right answer is "use BaseModel instead of dataclass", just wondering.

Running this

from dataclasses import dataclass
from pydantic import validate_arguments, BaseModel

@validate_arguments
@dataclass
class TestyClass:
    '''Section "F2 - Processing parameters"
        which isn't referenced in format.temp
    '''
    stringy: str
    inty: int
    floaty: float
        
    @classmethod
    def from_dict(cls, d):
        return cls(
            stringy=d['s'],
            inty=d['i'],
            floaty=d['f']
        )

TestyClass.from_dict({'s':'piff', 'i': 1, 'f': 2})

throws

...
---> 22 TestyClass.from_dict({'s':'piff', 'i': 1, 'f': 2})

TypeError: 'classmethod' object is not callable

[dmontagu]

@jnj16180340 @validate_arguments is not meant to be put on classes, I think what you'd want is just

from pydantic.dataclasses import dataclass

@dataclass
class TestyClass:
     ...

Also, it seems to work properly if you put @validate_arguments under @classmethod:

    @classmethod
    @validate_arguments
    def from_dict(cls, d):
        return cls(
            stringy=d['s'],
            inty=d['i'],
            floaty=d['f']
        )

(It does seem unfortunate that you can't put it outside/"above" the @classmethod, maybe we can fix that in v2...)

[malikoth]

This may be scope change, but I'd love to at least hear your thinking on this. Function signatures (optionally) contain one more type annotation that validate_arguments does not currently touch at all, namely the return type annotation. I would love to be able to use this not only on the data coming into a function, but also on the data going out.

For example:

import pydantic

@pydantic.validate_arguments
def foo(a: float) -> int:
    return a * 2

foo(1)

This currently returns 2.0, but since I've explicitly annotated the return type of the function as being int, I'd expect the returned value to be 2.

I wrote an implementation of this before I found validate_arguments, and the only thing I had to adjust with regards to return type was that if the return type is None, you don't want to try to convert return values to NoneType (or type(None)) you just want to replace the value with None.

[malikoth]

Update: I see that you CAN now validate return values in v2! Huzzah!

[samuelcolvin]

Closing this as we have no plans to make further big changes, if you want specific things, please create a new issue.