Fix DAG run state not updated while DAG is paused #16343
Conversation
ephraimbuddy
requested review from
ashb,
kaxil,
turbaszek and
XD-DENG
as code owners
github-actions
bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
from
25af23f
to
cc76e09
Compare
|
I wonder what is more efficient: doing this periodically (for paused dags, where the state is likely to never change) or expanding on the "mini scheduler run" to do a simpler version of |
Nice but I think it may not work if the user disables mini scheduling? |
Yes, but we'll likely remove that setting in a version or two -- it was mostly an escape hatch in case it had un-forseen bugs. |
Should I add it as a separate check outside the mini scheduling? |
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
4 times, most recently
from
1a941b1
to
5f386be
Compare
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
2 times, most recently
from
0fcc3f1
to
8d420f1
Compare
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
2 times, most recently
from
a06e51e
to
3bf9b22
Compare
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
from
3bf9b22
to
0c8d695
Compare
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
from
1ee7b0f
to
17b92bd
Compare
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
from
17b92bd
to
263767a
Compare
ephraimbuddy
force-pushed
the
update-dagrun-state-dag-paused
branch
2 times, most recently
from
859f92b
to
6e41893
Compare
The state of a DAG run does not update while the DAG is paused. The tasks continue to run if the DAG run was kicked off before the DAG was paused and eventually finish and are marked correctly. The DAG run state does not get updated and stays in Running state until the DAG is unpaused. This change fixes it by running a check on task exit to update state(if possible) of the DagRun if the task was able to finish the DagRun while the DAG is paused Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 3834df6)
Closes: #15439
The state of a DAG run does not update while the DAG is paused.
The tasks continue to run if the DAG run was kicked off before
the DAG was paused and eventually finish and are marked correctly.
The DAG run state does not get updated and stays in Running state until the DAG is unpaused.
This change fixes it by running a check at intervals, updating states(if possible)
of DagRuns that the tasks have finished running while the DAG is paused
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.