Cannot parse nested object with custom root type

[sohama4]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and couldn't find an answer
• After submitting this, I commit to one of:
  • Look through open issues and helped at least one other person
  • Hit the "watch" button on this repo to receive notifications and I commit to help at least 2 people that ask questions in the future
  • Implement a Pull Request for a confirmed bug

Question

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.7.2
            pydantic compiled: True
                 install path: <path>/venv/lib/python3.7/site-packages/pydantic
               python version: 3.7.7 (default, Nov  3 2020, 00:02:20)  [Clang 12.0.0 (clang-1200.0.32.21)]
                     platform: Darwin-19.6.0-x86_64-i386-64bit
     optional deps. installed: []

Hello, thank you for the library - I have used it in production across multiple applications now! I am trying to parse a JSON with a nested JSON needing a custom root type of Dict - but get an error as shown in the snippet. I tried to check thru old bugs and the fixes for them like #1253 and #1773 and verified that all fixes are already in the version of the code I have pulled, but still don't see the behavior fixed. Can someone help point what I'm doing wrong?

Thanks!

import pydantic
import json

class Player(BaseModel):
    player: str
    number: int
    pos: str

class Lineup(BaseModel):
    coach: str
    startXI: List[Player]
    substitutes: List[Player]


class Lineups(BaseModel):
    __root__: Dict[str, Lineup]


class LineupsResponse(BaseModel):
    results: int
    lineUps: Lineups


class LineupsApiResponse(BaseModel):
    api: LineupsResponse

if __name__ == "__main__":
    text = '{"api":{"results":2,"lineUps":{"ManchesterCity":{"coach":"Guardiola","coach_id":4,"formation":"4-3-3","startXI":[{"team_id":50,"player_id":617,"player":"Ederson","number":31,"pos":"G"}],"substitutes":[{"team_id":50,"player_id":50828,"player":"ZackSteffen","number":13,"pos":"G"}]},"Liverpool":{"coach":"J.Klopp","coach_id":1,"formation":"4-2-2-2","startXI":[{"team_id":40,"player_id":280,"player":"Alisson","number":1,"pos":"G"}],"substitutes":[{"team_id":40,"player_id":18812,"player":"Adrián","number":13,"pos":"G"},{"team_id":40,"player_id":127472,"player":"NathanielPhillips","number":47,"pos":"D"},{"team_id":40,"player_id":293,"player":"CurtisJones","number":17,"pos":"M"}]}}}}'
    try:
        response: LineupsApiResponse = LineupsApiResponse.parse_obj(json.loads(text))
    except Exception as e:
        print(e)
    try:
        response2: LineupsApiResponse = LineupsApiResponse.parse_raw(text)
    except Exception as e:
        print(e)

Output:

1 validation error for LineupsApiResponse
api -> lineUps -> __root__
  field required (type=value_error.missing)
1 validation error for LineupsApiResponse
api -> lineUps -> __root__
  field required (type=value_error.missing)

[PrettyWood]

Hello @sohama4
You're right __root__ is only supported at parent level and is not handled properly for nested models.

To support this we could add this logic directly

• in BaseModel.validate but it wouldn't work with construct for example
• in BaseModel.__init__ but it would cost a bit of running time

I run the benchmark and it doesn't seem to have any effect on the performance so the latter solution could be a good solution. @samuelcolvin WDYT?

If you want to try it out already @sohama4 I would go with something like this

import json
from typing import Any, Dict, List

from pydantic import BaseModel as PydanticBaseModel
from pydantic.utils import ROOT_KEY


class BaseModel(PydanticBaseModel):
    def __init__(__pydantic_self__, **data: Any) -> None:
        if __pydantic_self__.__custom_root_type__ and data.keys() != {ROOT_KEY}:
            data = {ROOT_KEY: data}
        super().__init__(**data)

...

Hope it helps!

[sohama4]

@PrettyWood - thank you for the response, I will try it out soon, and will close this out ASAP. I think the docs must mention this behavior, so I left a comment on the PR you linked.

[scorpp]

Similar issue with respect to .from_orm(). Solution advised by @PrettyWood doesn't work for this case since validate_model is called before constructor.

Worked this around like this:

class BaseCustomRootModel(BaseModel):

    @classmethod
    def from_orm(cls, obj: Any):
        if cls.__custom_root_type__:
            class Wrapper:
                __slots__ = ('__root__',)

                def __init__(self, obj):
                    self.__root__ = obj

            return super().from_orm(Wrapper(obj))

        return super().from_orm(obj)

             pydantic version: 1.7.3
            pydantic compiled: False
                 install path: /home/user/.local/share/virtualenvs/project-hash/lib/python3.6/site-packages/pydantic
               python version: 3.6.8 (default, Feb 11 2019, 08:59:55)  [GCC 8.2.1 20181127]
                     platform: Linux-5.9.11-zen2-1-zen-x86_64-with-arch
     optional deps. installed: ['email-validator']

[PrettyWood]

@scorpp What about this solution? It should work with everything above + orm if I'm not mistaken
So something like this

from pydantic import BaseModel as PydanticBaseModel

class BaseModel(PydanticBaseModel):
    @classmethod
    def validate(cls, value):
        if cls.__custom_root_type__:
            return cls.parse_obj(value)
        return super().validate(value)

[scorpp]

@PrettyWood no, this doesn't work. in BaseModel.from_orm in main.py:570 m = cls.__new__(cls) returns a model with no __root__ set. and then the call to validate_model on next line fails.

Also your option with overriding validate turn execution to different path - parse_obj instead of from_orm. Meaning that nested ORM models won't work anyway, since parse_obj will not accept regular objects.

I think that my solution (or alike) should be incorporated into from_orm of the BaseModel. Similarly to parse_obj.

[PrettyWood]

Ah yes sorry I wrote that on my phone yesterday night and should have paid more attention on the from_orm implementation.
I guess the fix is trivial if you can directly change the code like #2237
WDYT @scorpp ?

[samuelcolvin]

I've merged both #2237 and #2238, thanks so much @PrettyWood for working on this.

I think we can simplify the interface as well as improve performance in v2, but these two fixes should provide all the functionality required here.