How does DAG deletion work?

[CatarinaSilva]

I've been trying to understand DAG deletion, in particular how much gets removed. I have a few questions that I'm chasing, if anyone can give me a hand:

• DAGs can be removed explicitly in the UI or they can disappear if the files get removed. Do both processes trigger the same deletion or do they differ in any way?
• When a DAG gets removed, are DAG runs still kept in the db? Or are they fully removed?
• If we remove a file accidentally and then put the file back, can we have loss of data?
• Do logs get explicitly removed when a DAG gets removed? If so, is this the same for both "local" and "cloud" logs (eg. s3)?

[potiuk]

The DAGs have several states when it comes to being "not running". DAGs can be paused, deactivated
and finally all metadata for the DAG can be deleted.

Dag can be paused via UI when it is present in the DAGS_FOLDER, and scheduler stored it in
the database, but the user chose to disable it via the UI. The "pause" and "unpause" actions are available
via UI and API. Paused DAG is not scheduled by the Scheduler, but you can trigger them via UI for
manual runs. In the UI, you can see Paused DAGs (in Paused tab). The DAGs that are un-paused
can be found in the Active tab.

Dag can be deactivated (do not confuse it with Active tag in the UI by removing them from the
DAGS_FOLDER. When scheduler parses the DAGS_FOLDER and misses the DAG that it had seen
before and stored in the database it will set is as deactivated. The metadata and history of the
DAG` is kept for deactivated DAGs and when the DAG is re-added to the DAGS_FOLDER it will be again
activated and history will be visible. You cannot activate/deactivate DAG via UI or API, this
can only be done by removing files from the ``DAGS_FOLDER``. Once again - no data for historical runs of the
DAG are lost when it is deactivated by the scheduler. Note that the ``Active`` tab in Airflow UI
refers to DAGs that are both ``Activated`` and ``Not paused`` so this might initially be a
little confusing.

You can't see the deactivated DAGs in the UI - you can sometimes see the historical runs, but when you try to
see the information about those you will see the error that the DAG is missing.

You can also delete the DAG metadata from the metadata database using UI or API, but it does not
always result in disappearing of the DAG from the UI - which might be also initially a bit confusing.
If the DAG is still in DAGS_FOLDER when you delete the metadata, the DAG will re-appear as
Scheduler will parse the folder, only historical runs information for the DAG will be removed.

This all means that if you want to actually delete a DAG and its all historical metadata, you need to do
it in three steps:

• pause the DAG
• delete the historical metadata from the database, via UI or API
• delete the DAG file from the DAGS_FOLDER and wait until it becomes inactive

This information was missing in our docs, so I am adding it now.

[potiuk]

Updated it slightly to be more accurate.

[nicnguyen3103]

Hi @potiuk , I have encountered a very interesting case when dag is in Deactivated:

1. Remove the Dag from the DagBag while the dag is still running. There are some task in the schedule or running state
2. Task won't be able to complete, and upon refresh, Airflow complains about the Dag went missing.
3. However the status of the Dag on the UI is still running. Upon adding back the Dag to DagBag, all the task resume, and continue running as normal.

My question are:

1. If a dag is running (not paused) and there are tasks in the running state, upon removing the dag from dagbag, how would Airflow handle these task? Will it gracefully send sigterm to these tasks, or these tasks become zombies?
2. Should the dag status metadata switch to paused after being removed from DagBag? Because reintroducing the dag back to the DagBag will resume all the previous job, which may trigger an undesirable backfill process.

Many thanks for your help and have a nice day!

[potiuk]

No idea, really. DAG versioning is not implemented in Airflow and changing it's definition during running might have undesirable and random side-effects. Just don't do it.

[satyamsareen]

Shouldn't this line

"Note that the Active tab in Airflow UI
refers to DAGs that are not both Activated and Not paused so this might initially be a
little confusing."

be saying

"Note that the Active tab in Airflow UI
refers to DAGs that are both Activated and Not paused so this might initially be a
little confusing."

Dags under the active tab are both activated and not paused right?

[potiuk]

Sure. Obvious typo.

[SbtrktCtrl]

[potiuk]

Not sure what you are asking for. Can you be more precise what's your base of asking the questions and whether you can share some examples showing some problems? It's hard to elaborate on an open-ended question like this, but if you have a concrete example of problem you are dealing with, I invite you to open a new dicussion thread about it with all the details.

[DimitarSirakov]

I have a problem, i deleted a dag from the UI and now i can't recover it, i did try adding new line, changing the dag id, changing the file name, i checked the dag folder and the dag file is present, how can i recover the dag ? I didn't find any documentation about this.