Retry even after task was success

[Raul824]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

We have setup retry in case of failures and disconnects on individual task.
Some tasks are being reran post completion.

If a task is success it should not attempt to rerun it again.

At 8:05 the task was completed marked success by airflow.

[2023-12-04, 08:05:23 AEDT] {taskinstance.py:1350} INFO - Marking task as SUCCESS. dag_id=UDPPRDAU_ODS_KEY_SCD_SCF_06, task_id=SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL, execution_date=20231202T125900, start_date=20231203T210522, end_date=20231203T210523

But At 8:42 it again reran the same task and there are no clear events on this task in audit log.

*** Found local files:
*** * /opt/airflow/logs/dag_id=UDPPRDAU_ODS_KEY_SCD_SCF_06/run_id=manual__2023-12-02T12:59:00+00:00/task_id=SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL/attempt=3.log
[2023-12-04, 08:42:20 AEDT] {taskinstance.py:1097} INFO - Dependencies not met for <TaskInstance: UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL manual__2023-12-02T12:59:00+00:00 [success]>, dependency 'Task Instance State' FAILED: Task is in the 'success' state.
[2023-12-04, 08:42:20 AEDT] {local_task_job_runner.py:154} INFO - Task is not able to be run
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL manual__2023-12-02T12:59:00+00:00 [queued]>
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL manual__2023-12-02T12:59:00+00:00 [queued]>
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1308} INFO - Starting attempt 3 of 501
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1327} INFO - Executing <Task(PythonOperator): SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL> on 2023-12-02 12:59:00+00:00
[2023-12-04, 09:45:29 AEDT] {standard_task_runner.py:57} INFO - Started process 1128 to run task

My understanding was that retries comes into picture only in case of failures, which working fine in most of the tasks but it randomly reruns the tasks even after success.

What you think should happen instead

Once the task is success, airflow should not retry the task.

How to reproduce

Set up_for_retry for all tasks and run 200 tasks at the same time.

Operating System

Azure Kubernetes Services

Versions of Apache Airflow Providers

Airflow 2.6.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Raul824]

Below is the snippet of another such instance.
All tasks in a task group are sequential with trigger rule all_success.
max active run of dag is 1 and depends on past is true.

The dag is success even the task CMMS_DIGT_ELY is successful in logs but it went in up_for_retry status without any external interference. The next dag started running as soon as the previous one was success but task went in up_for_retry and due to depends on past as True it was stuck.

[eladkal]

Coverting to discussion as you didnt add reproduce example.
Please share reproduce example

[mkavakebi]

I have the same Problem. the first try is successfull, but still it starts the second try.
in another word it randomly runs two times and both are marked as success

[potiuk]

How about providing more details, logs and describing your deployment? Without it, it does not bring anyone closer to being able to diagnose it.

[Raul824]

Sorry, but I don't have any steps to replicate it yet. I would have built a workaround to handle the exceptions if I would have been able to replicate it.

So far the only thing I know is that even after success the postgres table was still showing the task as up_for_retry in task_instance. So my best guess would be that some issue in status update in postgres database.