Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor cloudpickle support in Python operators/decorators #39270

Merged
merged 6 commits into from May 4, 2024

Conversation

Taragolis
Copy link
Contributor

Some refactor cloudpickle support which added into the #38531

Use new serializer parameter for specify serialiser, this one for avoid boolean trap. Deprecate use_dill parameter in favor of serializer="dill" and remove use_cloudpickle in favor of serializer="cloudpickle", this one safe to remove because change not released yet.

Fix situation if cloudpickle installed then dill serialisation not available and implicitly use cloudpickle instead.

if shutil.which("cloudpickle") or importlib.util.find_spec("cloudpickle"):
import cloudpickle as serialization_library
elif shutil.which("dill") or importlib.util.find_spec("dill"):
import dill as serialization_library
else:
log.debug("Neither dill and cloudpickle are installed. Please install one with: pip install [name]")

In addition lazy load serialisers as late as possible, for avoid the situation that we load all serialiser even if users do not use any of them

cc @VladaZakharova


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis added full tests needed We need to run full set of tests for this PR to merge all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Apr 25, 2024
@Taragolis Taragolis added this to the Airflow 2.10.0 milestone Apr 25, 2024
@boring-cyborg boring-cyborg bot added the area:core-operators Operators, Sensors and hooks within Core Airflow label Apr 25, 2024
@Taragolis Taragolis force-pushed the refactor-cloupickle-support branch 2 times, most recently from 680e0a2 to b23384d Compare April 30, 2024 20:23
@Taragolis Taragolis force-pushed the refactor-cloupickle-support branch from b23384d to a2e2bb2 Compare May 3, 2024 12:45
@Taragolis Taragolis merged commit 7b5962e into apache:main May 4, 2024
93 checks passed
@Taragolis Taragolis deleted the refactor-cloupickle-support branch May 4, 2024 18:34
RodrigoGanancia pushed a commit to RodrigoGanancia/airflow that referenced this pull request May 10, 2024
…9270)

* Refactor cloudpickle support in Python operators/decorators

* Fixup missing marker

* Return back skip TestPythonVirtualenvOperator::test_airflow_context for dill

* TestPythonVirtualenvOperator::test_airflow_context xfail instead of skip

* Catch only on ModuleNotFound error and simple reraise with warning

* Limit test_airflow_context only for python 3.11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs area:core-operators Operators, Sensors and hooks within Core Airflow full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants