RuntimeError: dictionary changed size during iteration #685
Comments
|
Can you share the models so I can try to reproduce the error? |
|
Yes, thank you 🙂 import os
from enum import Enum as StdEnum
from typing import List, Union
import databases
import ormar
import pydantic
import sqlalchemy
MAX_LIBRARY_NAME_LENGTH: int = 100
MAX_LIBRARY_KIND_LENGTH: int = 16
MAX_LIBRARY_SUMMARY_LENGTH: int = 1000
MAX_LIBRARY_URL_LENGTH: int = 1000
MAX_PACKAGE_VERSION_LENGTH: int = 64
MAX_PROJECT_APP_CODE_LENGTH: int = 16
MAX_PROJECT_NAME_LENGTH: int = 100
MAX_PROJECT_PLATFORM_LENGTH: int = 64
MAX_PROJECT_DEPENDENCY_GIT_BRANCH_LENGTH: int = 128
MAX_PROJECT_DEPENDENCY_USAGE_LENGTH: int = 1000
MAX_TEAM_NAME_LENGTH: int = 16
MAX_TICKET_KIND_LENGTH: int = 16
MAX_TICKET_STATUS_LENGTH: int = 16
MAX_TICKET_PACKAGE_COMMENT_LENGTH: int = 1000
MAX_TICKET_PACKAGE_STATUS_LENGTH: int = 16
class Enum(StdEnum):
@classmethod
def values(cls) -> List[Union[str, int]]:
return [item.value for item in cls]
class LibraryKind(Enum):
pypi: str = "pypi"
nuget: str = "nuget"
maven: str = "maven"
npm: str = "npm"
docker: str = "docker"
helm: str = "helm"
rpm: str = "rpm"
generic: str = "generic"
class TicketPackageStatus(Enum):
in_progress: str = "in progress"
accepted: str = "accepted"
rejected: str = "rejected"
class TicketStatus(Enum):
open: str = "open"
closed: str = "closed"
class TicketKind(Enum):
monitoring: str = "monitoring"
analysis: str = "analysis"
urbanization: str = "urbanization"
security: str = "security"
class BaseMeta(ormar.ModelMeta):
database = databases.Database("sqlite:///db.sqlite", timeout=30)
metadata = sqlalchemy.MetaData()
class Team(ormar.Model):
class Meta(BaseMeta):
tablename = "teams"
constraints = [ormar.UniqueColumns("name")]
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=MAX_TEAM_NAME_LENGTH)
@pydantic.validator("name")
def _name_must_be_uppercase(cls, value): # noqa: N805
return value.upper()
class Library(ormar.Model):
class Meta(BaseMeta):
tablename = "libraries"
constraints = [ormar.UniqueColumns("name", "kind")]
id: int = ormar.Integer(primary_key=True)
name: str = ormar.String(max_length=MAX_LIBRARY_NAME_LENGTH)
kind: str = ormar.String(max_length=MAX_LIBRARY_KIND_LENGTH, choices=LibraryKind.values())
summary: str = ormar.String(max_length=MAX_LIBRARY_SUMMARY_LENGTH)
url: str = ormar.String(max_length=MAX_LIBRARY_URL_LENGTH)
@pydantic.root_validator
def _normalize_name(cls, values): # noqa: N805
kind = values.get("kind")
if kind is LibraryKind.pypi:
values["name"] = values["name"].lower().replace("_", "-")
return values
class Package(ormar.Model):
class Meta(BaseMeta):
tablename = "packages"
constraints = [ormar.UniqueColumns("library", "version")]
id: int = ormar.Integer(primary_key=True)
library: Library = ormar.ForeignKey(Library, related_name="packages", ondelete="CASCADE")
version: str = ormar.String(max_length=MAX_PACKAGE_VERSION_LENGTH)
class Ticket(ormar.Model):
class Meta(BaseMeta):
tablename = "tickets"
constraints = [ormar.UniqueColumns("number")]
id: int = ormar.Integer(primary_key=True)
number: int = ormar.Integer()
kind: TicketKind = ormar.String(max_length=MAX_TICKET_KIND_LENGTH, choices=TicketKind.values())
status: TicketStatus = ormar.String(max_length=MAX_TICKET_STATUS_LENGTH, choices=TicketStatus.values())
team: Team = ormar.ForeignKey(Team, related_name="tickets", ondelete="CASCADE")
class TicketPackage(ormar.Model):
class Meta(BaseMeta):
tablename = "tickets_packages"
constraints = [ormar.UniqueColumns("ticket", "package")]
id: int = ormar.Integer(primary_key=True)
status: TicketPackageStatus = ormar.String(
max_length=MAX_TICKET_PACKAGE_STATUS_LENGTH, choices=TicketPackageStatus.values()
)
comment: str = ormar.String(max_length=MAX_TICKET_PACKAGE_COMMENT_LENGTH)
ticket: Ticket = ormar.ForeignKey(Ticket, related_name="packages", ondelete="CASCADE")
package: Package = ormar.ForeignKey(Package, related_name="tickets", ondelete="CASCADE")
explicit: bool = ormar.Boolean(default=True) |
|
Happening on another query as well, here's the most verbose output I can get with pytest (it still truncates data...): We can see that in Query is: await Team.objects.filter(projects__dependencies__package__id=package_id).all()(I didn't provide these models above) Fixed with: await Team.objects.select_related("projects__dependencies__package__library").filter(projects__dependencies__package__id=package_id).all() |
|
I've ran a failing snippet with multiple versions of Ormar, downgrading from 0.10.25 to 0.10.0, and the snippet stopped failing at 0.10.23, so the "bug" was probably introduced in 0.10.24, by either Ormar itself or an upgraded version of one of its dependencies 🙂
The most suspicious dependency if of course pydantic since it appears in the traceback. I'll try to git bisect between 0.10.24 and 0.10.23. |
|
Possibly related: pydantic/pydantic@952fad2, from issue pydantic/pydantic#3641 |
|
Thanks for the investigation! Yep, it seems it's pydantic related as I didn't introduce anything that I can think of that could cause this error in 0.10.24. |
|
Just finished bisecting, and it confirms our suspicions: So yeah I think that the change from shallow to deep copy in pydantic broke Ormar 🙂 |
|
I can see that it also causes additional errors in prs open by dependabot in other places. Yeah, I need to check, I already overwritten some pydantic methods (dict, json etc.) in ormar so might need to add some more 😅 |
|
Should be fixed in 0.11.1. |
|
Awesome, thanks a lot! I'll be able to try that tomorrow, I'll let you know 🙂 |
|
Yep, everything is working fine again 🙂 Thanks! |
|
thank you |
|
Hi, I have the same problem. I was on ORMAR 0.10.25 and uploaded to 0.11.2 to get the fix and it keeps crashing. I think my case is slightly different, or more complex. The relationships within my model to other models work, the problem appears when in the third level I need a JOIN on the same model (the initial one). This code worked a few months ago. Example that works: Example that fails: Real code: Thanks in advance and congratulations on this awesome ORM! |
Describe the bug
Not sure Ormar will be able to do anything.
To Reproduce
Upon running a query, there's some Pydantic validation happening, and it ends up with a runtime error when deep-copying data.
It seems that it is due to an element (of the model being deep-copied) being a submodel, and that upon accessing it, or another attribute of the parent model, it is loaded, changing the dictionary size somehow.
Here's the query:
That query triggers the runtime error mentioned above. If I either remove
tickets__ticket, or go down to the submodel withtickets__ticket__team, the error disappears.Traceback:
Expected behavior
No runtime error.
Versions (please complete the following information):
ormarversion: 0.11.0pydanticversion: 1.9.1fastapiversion: 0.78.0Additional context
The code triggering the error did not always trigger it. It might be due to a dependency upgrade (which one? I don't know).
Did anyone encounter the same issue?
The text was updated successfully, but these errors were encountered: