Skip to content

Commit

Permalink
Disable attrs state management on MappedOperator
Browse files Browse the repository at this point in the history
The custom __getstate__ and __setstate__ implementation from attrs
interacts badly with Airflow's DAG serialization and pickling. When a
mapped task is deserialized, subclasses are coerced to MappedOperator.
But when the instances go through DAG pickling, all attributes defined
in the subclasses are dropped by attrs's custom state management.

Since attrs does not do anything too special here (the logic is only
important for slot=True), we can use Python's built-in implementation
instead.
  • Loading branch information
uranusjr committed Jul 1, 2022
1 parent 97948ec commit 3af6579
Showing 1 changed file with 11 additions and 1 deletion.
12 changes: 11 additions & 1 deletion airflow/models/mappedoperator.py
Expand Up @@ -240,7 +240,17 @@ def _expand(self, **mapped_kwargs: "Mappable") -> "MappedOperator":
return op


@attr.define(kw_only=True)
@attr.define(
kw_only=True,
# Disable custom __getstate__ and __setstate__ generation since it interacts
# badly with Airflow's DAG serialization and pickling. When a mapped task is
# deserialized, subclasses are coerced into MappedOperator, but when it goes
# through DAG pickling, all attributes defined in the subclasses are dropped
# by attrs's custom state management. Since attrs does not do anything too
# special here (the logic is only important for slots=True), we use Python's
# built-in implementation, which works (as proven by good old BaseOperator).
getstate_setstate=False,
)
class MappedOperator(AbstractOperator):
"""Object representing a mapped operator in a DAG."""

Expand Down

0 comments on commit 3af6579

Please sign in to comment.