Union Field: union_mode="smart" with default_factory and self reference ignores data that could be coerced

[guillaumegenthial]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

The union_mode default smart gets confused if

• the attribute is a self referencing union
• the raw data needs to be coerced
• the Field has a default factory

In this situation, it ends up picking the default factory even if it should be capable of coercing the input.

NOTE: it seems that if you remove the default factory or don't use a self-referencing union, things work fine.

Example Code

from typing import Union

from pydantic import BaseModel, Field


class Point(BaseModel):
    x: int
    y: int


class Points(BaseModel):
    # This works
    # points: Union[Point, "Points"] = Field(
    #     union_mode="smart",
    # )

    # This ignores the data
    points: Union[Point, "Points"] = Field(
        union_mode="smart",
        default_factory=lambda: Point(x=0, y=0),
    )


points = Points(
    points={
        "x": "1",
        "y": "1",
    },
)
print(points)  # points=Points(points=Point(x=0, y=0))


points = Points(
    points={
        "x": 1,
        "y": 1,
    },
)
print(points)  # points=Point(x=1, y=1)

Python, Pydantic & OS Version

pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=true
                 install path: .../envs/fragment-3.11/lib/python3.11/site-packages/pydantic
               python version: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:38:07) [Clang 16.0.6 ]
                     platform: macOS-14.4-arm64-arm-64bit
             related packages: typing_extensions-4.9.0 mypy-1.8.0 fastapi-0.104.1 email-validator-2.1.0.post1
                       commit: unknown

Related

#2135
#2092

[guillaumegenthial]

To fix, one option is to specify union_mode=left_to_right (see Pydantic docs on Unions).

class Points(BaseModel):
    # This works
    # points: Union[Point, "Points"] = Field(
    #     union_mode="smart",
    # )

    # This ignores the data
    points: Union[Point, "Points"] = Field(
        union_mode="left_to_right",
        default_factory=lambda: Point(x=0, y=0),
    )

[guillaumegenthial]

However, if you have a list of Unions, you can't use this trick

from typing import Union

from pydantic import BaseModel, Field


class Point(BaseModel):
    x: int
    y: int


class Points(BaseModel):
    # This works
    # points: Union[Point, "Points"] = Field(
    #     union_mode="smart",
    # )

    # This ignores the data
    points: list[Union[Point, "Points"]] = Field(
        union_mode="left_to_right",
        default_factory=lambda: Point(x=0, y=0),
    )


points = Points(
    points=[
        {
            "x": "1",
            "y": "1",
        }
    ],
)
print(points)  # points=Points(points=Point(x=0, y=0))


points = Points(
    points=[
        {
            "x": 1,
            "y": 1,
        }
    ],
)
print(points)  # points=Point(x=1, y=1)

which raises

TypeError: The following constraints cannot be applied to list[typing.Union[__main__.Point, __main__.Points]]: 'union_mode'

[guillaumegenthial]

I found a way to achieve the desired behavior using a Discriminator

from typing import Annotated, Union

from pydantic import BaseModel, Discriminator, Field, Tag


class Point(BaseModel):
    x: int
    y: int


def points_discrimnator(v):
    if isinstance(v, list):
        return "Points"
    return "Point"


class Points(BaseModel):
  
    points: list[
        Annotated[
            Union[Annotated[Point, Tag("Point")], Annotated["Points", Tag("Points")]],
            Discriminator(
                points_discrimnator,
                custom_error_type="invalid_union_member",
                custom_error_message="Invalid union member",
                custom_error_context={"discriminator": "str_or_model"},
            ),
        ]
    ] = Field(
        default_factory=lambda: Point(x=0, y=0),
    )


points = Points(
    points=[
        {
            "x": "1",
            "y": "1",
        }
    ],
)
print(points)  # points=Point(x=1, y=1)


points = Points(
    points=[
        {
            "x": 1,
            "y": 1,
        }
    ],
)
print(points)  # points=Point(x=1, y=1)

But it seems really overkill, is there a better way to achieve this ?

[sydney-runkle]

However, if you have a list of Unions, you can't use this trick

You can use Annotated on the Union within the List in order to apply this constraint via the Field(union_mode='left_to_right') annotation.

Given that, I think this issue is resolved...

[guillaumegenthial]

@sydney-runkle thanks for the quick reply.

Indeed this works

class Points(BaseModel):
    # This works
    points: list[
        Annotated[Union[Point, "Points"], Field(union_mode="left_to_right")]
    ] = Field(default_factory=list)

[guillaumegenthial]

Given that, I think this issue is resolved...

I would argue that it's not quite resolved. There is a workaround though.

Why ?

1. The behavior is not consistent (adding a default_factory should not change the resolved type ?)
2. It's unexpected (data that would otherwise be coerced is ignored)

This caused major issues on our product FYI and was quite hard to track down...

[sydney-runkle]

@guillaumegenthial,

Ah yes you're right. I didn't understand the issue in its entirety! Thanks for following up.