Disable attrs state management on MappedOperator #24772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The custom __getstate__ and __setstate__ implementation from attrs interacts badly with Airflow's DAG serialization and pickling. When a mapped task is deserialized, subclasses are coerced to MappedOperator. But when the instances go through DAG pickling, all attributes defined in the subclasses are dropped by attrs's custom state management. Since attrs does not do anything too special here (the logic is only important for slot=True), we can use Python's built-in implementation instead.
uranusjr
force-pushed
the
fix-mappedoperator-serialized-pickling
branch
from
a0168ef
to
3af6579
Compare
|
Add something to one of the round trip tests in |
ashb
approved these changes
Jul 5, 2022
|
Though I am surprised that we hit pickle anywhere in the AIP-45 change, since (from a cursor look at the code) it uses the existing serialized tables, |
|
It might not be pickling per-se (I still have not found the actual direct cause to the issue), but the same marshalling logic (that uses |
ephraimbuddy
pushed a commit
that referenced
this pull request
Aug 19, 2022
(cherry picked from commit 6fd06fa)
53 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The custom
__getstate__and__setstate__implementation from attrs interacts badly with Airflow's DAG serialization and pickling. When a mapped task is deserialized, subclasses are coerced to MappedOperator. But when the instances go through DAG pickling, all attributes defined in the subclasses are dropped by attrs's custom state management.Since attrs does not do anything too special here (the logic is only important for
slots=True), we can use Python's built-in implementation instead.I locally ran
airflow schedulerand this seems to fix the exceptions, but it seems difficult to create a meaningful test for this due to all the components involved. Ideas on test setup are much welcomed.Fix #23838.