Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add @task.short_circuit TaskFlow decorator #25752

Merged
merged 1 commit into from Sep 7, 2022

Conversation

josh-fell
Copy link
Contributor

@josh-fell josh-fell commented Aug 17, 2022

To complete the "Python operator" set, this PR adds a TaskFlow decorator abstraction for the ShortCircuitOperator.

TODO:

  • Add unit tests

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@josh-fell josh-fell marked this pull request as ready for review August 18, 2022 05:01
@uranusjr
Copy link
Member

A documentation should be added somewhere (docstring? TaskFlow howto?) to describe what returning True and False means.

@josh-fell
Copy link
Contributor Author

A documentation should be added somewhere (docstring? TaskFlow howto?) to describe what returning True and False means.

That's fair. I think updating the docs somewhere would be my preference. Maybe in the Python Operator how-tos doc?

Side note, maybe we should add a TaskFlow Decorators section to the Python API Reference in a separate PR. WDYT?

@potiuk
Copy link
Member

potiuk commented Aug 22, 2022

A documentation should be added somewhere (docstring? TaskFlow howto?) to describe what returning True and False means.

That's fair. I think updating the docs somewhere would be my preference. Maybe in the Python Operator how-tos doc?

Sounds good.

Side note, maybe we should add a TaskFlow Decorators section to the Python API Reference in a separate PR. WDYT?

Agree. Also - since some of the decorators are coming from the providers, we should add a section in https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html (I can add it and also refactor this page a bit as it is a bit too long now and I think we should make it more compact).

How about also reversing (in separate PR) the "priority" of "classic" vs. "taskflow" approach ? I think we are quickly approaching the point where taskflow is generally recommended and "on-par" approach than Classic and we should promote it as "the" way of doing things while marking classic approach as somewhat (not entirely but somewhat) legacty.

@josh-fell
Copy link
Contributor Author

Update the Python operator how-to doc with references to the new decorator.

@josh-fell
Copy link
Contributor Author

The test failures don't look related to this PR, but they do look to be resolved by reverting #26175 (at least locally).

@josh-fell josh-fell force-pushed the short-circuit-taskflow-deco branch 2 times, most recently from e490430 to 5ed5ca1 Compare September 6, 2022 17:17
@josh-fell
Copy link
Contributor Author

josh-fell commented Sep 7, 2022

Mainly rebasing to pick up #26188 which should hopefully yield clean CI checks.

@jedcunningham jedcunningham merged commit ebef9ed into apache:main Sep 7, 2022
@jedcunningham jedcunningham added this to the Airflow 2.4.0 milestone Sep 7, 2022
@jedcunningham jedcunningham added the type:new-feature Changelog: New Features label Sep 7, 2022
@josh-fell josh-fell deleted the short-circuit-taskflow-deco branch September 7, 2022 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants