chore: Migrate Python projects from Pydantic v1 to v2 #14871

ahiuchingau · 2024-04-11T16:57:30Z

Overview

Let's try this again

Closes PLAT-326.

Test Plan

Changelog

Update our robot software (and everything that depends on it) from Pydantic v1 to Pydantic v2:

api
robot-server
shared-data
g-code-testing
hardware
server-utils
system-server
hardware-testing (re-lock only, depends on api)
abr-testing (re-lock only, depends on api)
performance-metrics (re-lock only, depends on shared-data)

buildroot changes in [todo]

Review requests

Risk assessment

SyntaxColoring

I'm about 60% through the changes so far. Mostly looks great, only a few comments and questions:

SyntaxColoring · 2024-04-11T17:06:50Z

api/src/opentrons/calibration_storage/ot2/models/v1.py

+DatetimeType = typing.Annotated[
+    datetime,
+    PlainSerializer(lambda x: x.isoformat(), when_used="json"),
+]
+


I'm inferring that this is replacing the old json_encoders config. Does something need to replace the old json_decoders config, to match?

SyntaxColoring · 2024-04-11T17:09:12Z

api/src/opentrons/calibration_storage/ot3/models/v1.py

+DatetimeType = typing.Annotated[
+    datetime,
+    PlainSerializer(lambda x: x.isoformat(), when_used="json"),
+]


api/src/opentrons/protocol_engine/clients/sync_client.py

SyntaxColoring · 2024-04-11T17:20:06Z

api/src/opentrons/protocol_engine/commands/command_unions.py

+# Each time a TypeAdapter is instantiated, it will construct a new validator and
+# serializer. To improve performance, TypeAdapters are instantiated once.
+# See https://docs.pydantic.dev/latest/concepts/performance/#typeadapter-instantiated-once
+CommandCreateAdatper: TypeAdapter[CommandCreate] = TypeAdapter(CommandCreate)  # type: ignore[arg-type]


Typo: Adatper instead of Adapter

What is this TypeAdapter doing?

SyntaxColoring · 2024-04-11T17:31:16Z

api/src/opentrons/protocol_engine/state/commands.py

@@ -230,7 +230,7 @@ def handle_action(self, action: Action) -> None:  # noqa: C901
            # request > command mapping, figure out how to type precisely
            # (or wait for a future mypy version that can figure it out).
            # For now, unit tests cover mapping every request type
-            queued_command = action.request._CommandCls.construct(


Can this be .model_construct, for performance reasons?

SyntaxColoring · 2024-04-11T17:38:44Z

shared-data/python/opentrons_shared_data/types.py

+class Vec3f(BaseModel, Generic[NumberType]):
+    """A 3D Vector."""
+
+    x: NumberType
+    y: NumberType
+    z: NumberType


The name Vec3f comes from "vector of 3 floats." If this is generic across other kinds of NumberType, let's rename it to just Vec3.

Since we're really pushing this deep down into common shared code, I'm worried that someone will come along later and add an = 0 default for something without realizing that it will affect a million things. Let's spell out that this is specifically for vectors where all elements are required, to hopefully prompt people to use a different type if that's not the behavior that they want.

Suggested change

class Vec3f(BaseModel, Generic[NumberType]):

"""A 3D Vector."""

x: NumberType

y: NumberType

z: NumberType

class Vec3f(BaseModel, Generic[NumberType]):

"""A 3D vector where all elements are required."""

x: NumberType

y: NumberType

z: NumberType

SyntaxColoring · 2024-04-11T17:41:18Z

shared-data/python/tests/pipette/test_validate_schema.py

+    assert loaded_model.channels == types.PipetteChannelType.NINETY_SIX_CHANNEL
+
+    model_dict = loaded_model.model_dump()
+    # each field should be the value of the enum class


SyntaxColoring

I'm caught up!

SyntaxColoring · 2024-04-12T15:19:28Z

api/src/opentrons/protocol_runner/legacy_command_mapper.py

Same as the other comment—for performance reasons, can we change the .construct()s to .model_construct()s instead of changing them to fully-validating __init__() calls?

robot-server/robot_server/log.txt

SyntaxColoring · 2024-04-12T15:50:12Z

robot-server/setup.py

-    "fastapi==0.99.1",
+    "fastapi>=0.100.0",
    "python-dotenv==1.0.1",
    "python-multipart==0.0.6",
-    "pydantic==1.10.12",
+    "pydantic>=2.0.0,<3",


Do these need to be pinned to a specific version like they were before?

Please don't pin to exact versions unless strictly necessary. See e.g. #11905

This is robot-server, which is not published on PyPI, so it should not affect what gets pulled in when you do pip install opentrons.

I'm a bit shaky on our packaging mechanisms, but I think this specific setup.py may not even affect what we install on robots. I'm sure we can make this less confusing somehow. But for the sake of not changing too many things in this PR, I think we'll keep the ==.

Conflicts: * api/Pipfile.lock (took edge, will need to re-lock) * api/src/opentrons/calibration_storage/deck_configuration.py * api/src/opentrons/cli/analyze.py * api/src/opentrons/protocol_api/core/engine/protocol.py * api/src/opentrons/protocol_engine/commands/calibration/calibrate_gripper.py * api/src/opentrons/protocol_engine/commands/calibration/calibrate_pipette.py * api/src/opentrons/protocol_engine/commands/command.py * api/src/opentrons/protocol_engine/commands/custom.py * api/src/opentrons/protocol_engine/types.py * api/src/opentrons/protocol_runner/legacy_command_mapper.py * api/tests/opentrons/cli/test_cli.py * api/tests/opentrons/protocol_engine/state/command_fixtures.py * api/tests/opentrons/protocol_engine/state/test_addressable_area_store.py * api/tests/opentrons/protocol_engine/state/test_geometry_view.py * api/tests/opentrons/protocol_runner/test_protocol_runner.py * robot-server/Pipfile * robot-server/Pipfile.lock (took edge, will need to re-lock) * robot-server/robot_server/deck_configuration/defaults.py * robot-server/robot_server/persistence/pydantic.py * shared-data/command/schemas/8.json (took edge, will need to regenerate) * shared-data/python/tests/deck/test_typechecks.py

For some reason, this is necessary for `pipenv lock --dev` to succeed. Otherwise, when it resolves performance-metrics' subdependencies, it mistakenly tries to get opentrons-shared-data from an external source instead of from this local path, and then complains because the external opentrons-shared-data's dependencies are incompatible with api's dependencies.

Pydantic doesn't allow this anymore.

…rons into chore_update-pydantic-v2

Forgot this in commit 86623ea.

Same reason as in commit 913019f.

Just for consistency with the existing pattern.

Handy dandy scripts: Find projects whose dependencies have changed: git diff --name-only edge | grep -E 'setup\.py|Pipfile$' Iterate over projects and re-lock them: for dir in shared-data/python server-utils hardware api system-server robot-server g-code-testing; do echo $dir && pushd $dir && pipenv lock --dev && popd; done

This has merge conflicts, so let's make sure we don't flagrantly break it.

Merging for updates to robot-server tests.

ahiuchingau added 14 commits April 11, 2024 13:01

update pydantic & fix all mypy errors

c2d17fb

hardware: update pydantic & fix all mypy errors

cae6028

hardware: fix Pipfile

8779c87

api: update pydantic & fix mypy errors

19f7110

shared-data: add Vec3f

b5c8eab

shared-data: rename labware_definition to models

ac77352

fix api mypy test errors

ef889d4

update api

b3ea918

fix protocol_engine tests

ea9e17e

fix(api): tests

90b7fab

update command schema 8.json

16b69eb

shared-data: deprecate json_encoder and use v2 serializer

528a825

fix api tests

ff83a57

api(src): update deprecated code

80a7000

ahiuchingau force-pushed the chore_update-pydantic-v2 branch from e83d1b5 to 80a7000 Compare April 11, 2024 17:03

ahiuchingau added 2 commits April 11, 2024 13:12

api: linter

1291aa1

shared-data: remove deprecated code

27bbbec

SyntaxColoring reviewed Apr 11, 2024

View reviewed changes

ahiuchingau added 6 commits April 11, 2024 13:51

shared-data: mypy.ini removed commented out override

b646339

robot-server & server-utils: update pipenv

7588a3f

robot-server: add pydantic_settings

0b391da

robot-server: bump-pydantic to v2

8114936

robot-server: fix lint errors

52b5899

robot-server: fix one test

9b02831

SyntaxColoring reviewed Apr 12, 2024

View reviewed changes

make lint

2728cca

SyntaxColoring mentioned this pull request Apr 15, 2024

feat: Upgrade pydantic to v2 #13983

Open

ahiuchingau force-pushed the chore_update-pydantic-v2 branch from 949e25a to 2728cca Compare April 17, 2024 18:50

ahiuchingau and others added 2 commits April 19, 2024 13:09

add model serializer

c47cb10

SyntaxColoring added 25 commits May 23, 2024 13:23

Update system-server Pipfile.

8ed4c1e

Re-lock api.

2938afb

_TestCommand.result and ReloadLabware.result need to default to None.

2b7d068

Revert SyncClient model_validate() changes.

ac00d02

Remove now-unneeded type-ignores.

10aa798

Don't instantiate BaseModel directly.

fef7108

Pydantic doesn't allow this anymore.

Merge branch 'chore_update-pydantic-v2' of github.com:Opentrons/opent…

a7a69d7

…rons into chore_update-pydantic-v2

Regenerate command schema.

46b024b

Delete empty log.txt.

7d73026

Update .json() calls in cli.analyze.

14bc816

Run bump-pydantic on api.

8dbe3d8

Give up on test_command_executor.py.

d975c0e

Pin fastapi with == like we had before.

e677a7e

Forgot this in commit 86623ea.

Ensure all fastapi pins consistently use 0.100.0.

3771b91

Move robot-server's dependency on performance-metrics to [packages].

ec14dd7

Same reason as in commit 913019f.

Install pydantic-settings in system-server.

d3c6ce2

Add pydantic-settings to robot-server/setup.py.

34cae8c

Just for consistency with the existing pattern.

robot-server: Run bump-pydantic.

1b3b099

robot-server: Remove now-unneeded type-ignores.

06e87fd

robot-server: Add a unit test for robot_server.persistence.pydantic.

9cdeb04

This has merge conflicts, so let's make sure we don't flagrantly break it.

robot-server: Post-merge fixups to robot_server.persistence.pydantic.

5f60e97

system-server: Run bump-pydantic.

dd461e9

system-server: Manual fixups.

3b1da95

SyntaxColoring mentioned this pull request May 28, 2024

chore(pydantic): V1->V2 migration #14805

Closed

Merge remote-tracking branch 'origin/edge' into chore_update-pydantic-v2

ff3db4e

Merging for updates to robot-server tests.

SyntaxColoring closed this May 28, 2024

SyntaxColoring deleted the chore_update-pydantic-v2 branch May 28, 2024 17:14

SyntaxColoring reopened this May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Migrate Python projects from Pydantic v1 to v2 #14871

chore: Migrate Python projects from Pydantic v1 to v2 #14871

ahiuchingau commented Apr 11, 2024 •

edited by SyntaxColoring

SyntaxColoring left a comment

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring left a comment

SyntaxColoring Apr 12, 2024

SyntaxColoring Apr 12, 2024

ecederstrand May 24, 2024 •

edited

SyntaxColoring May 24, 2024 •

edited

chore: Migrate Python projects from Pydantic v1 to v2 #14871

Are you sure you want to change the base?

chore: Migrate Python projects from Pydantic v1 to v2 #14871

Conversation

ahiuchingau commented Apr 11, 2024 • edited by SyntaxColoring

Overview

Test Plan

Changelog

Review requests

Risk assessment

SyntaxColoring left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SyntaxColoring left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecederstrand May 24, 2024 • edited

Choose a reason for hiding this comment

SyntaxColoring May 24, 2024 • edited

Choose a reason for hiding this comment

ahiuchingau commented Apr 11, 2024 •

edited by SyntaxColoring

ecederstrand May 24, 2024 •

edited

SyntaxColoring May 24, 2024 •

edited