Memory usage increases significantly going from pydantic 1.9.0 to 1.9.1

[mansoor2016]

Checks

• [X ] I added a descriptive title to this issue
• [X ] I have searched (google, github) for similar issues and couldn't find anything
• [X ] I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.9.1
            pydantic compiled: True
                 install path: /home/mansoor/source/test-oracle/venv/lib/python3.9/site-packages/pydantic
               python version: 3.9.13 (main, May 23 2022, 22:01:06)  [GCC 9.4.0]
                     platform: Linux-5.13.0-48-generic-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

Updating between pydantic 1.9.0 and 1.9.1 results in a dramatic spike in memory usage for our production application. This is most prominent where we have deep nested BaseModel classes serialising large amounts of data to json.

The memory profiler indicates that Pydantic 1.9.1 is deep copying dictionary data, which it was not doing before

Attempted MRE:
There is a ~40% increase in mem usage for this example

from typing import Generic, TypeVar, List

import orjson
import tempfile
from pydantic import BaseModel, Field
from pydantic.generics import GenericModel

_T = TypeVar('_T')


class MyGenericValue(GenericModel, Generic[_T]):
    value: _T


class GenericValueList(GenericModel, Generic[_T]):
    __root__: List[MyGenericValue[_T]]


class MyClass(BaseModel):
    n = 10000
    my_list: GenericValueList[float] = \
        Field(GenericValueList(__root__=[MyGenericValue(value=42.0)] * n))


if __name__ == '__main__':
    nested_base_model = MyClass()
    tdm_json = orjson.dumps(nested_base_model.dict())
    output_file = tempfile.NamedTemporaryFile()
    with open(output_file.name, "wb") as fh:
        fh.write(tdm_json)

[samuelcolvin]

Is this related to #3641?

How does the memory profile compare with v1.8.2?

Can you provide a self contained, minimal example?

[mansoor2016]

Is this related to #3641?

Doesn't look like its related

How does the memory profile compare with v1.8.2?

v1.8.2 is similar to v1.9.0, peak memory usage was 3GB.

Can you provide a self contained, minimal example?

Will try to generate one later.

[samuelcolvin]

opposite problem, but might be worth looking at #3829 too.

[mansoor2016]

I've have added a minimal reproducible example,

[kveretennicov]

@samuelcolvin, is this issue on some kind of roadmap? Cheers

[samuelcolvin]

Well without a clear culprit and even proposed solution, there's not much we can do for v1.10.

The entire design has changed completely for V2 so I would hope this is solved, but would love to see some memory profiling of pydantic-core if someone is willing to investigate?

[sydney-runkle]

We’re no longer actively developing Pydantic V1 (although it will continue to receive security fixes for the next year or so), so we’re closing issues solely related to V1. If you really think this shouldn’t be closed, please comment or create a new issue 🚀.