From 1463e635f7c73de81456ae1238edd85bb89cb913 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Tue, 8 Feb 2022 14:58:05 -0600 Subject: [PATCH 01/11] added explaining concept of logical date in DAG run Added two paragraphs under the DAG run to describe the difference between logical date and dag start and end date, and some of the reason behind its naming 'logical,' as well as providing an example. --- docs/apache-airflow/concepts/dags.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 3edaf35d5759b..9cae2fbd09bca 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -177,6 +177,19 @@ In much the same way a DAG instantiates into a DAG Run every time it's run, Tasks specified inside a DAG are also instantiated into :ref:`Task Instances ` along with it. +When the DAG run starts, it will have the start date and end date. This period +describes the time when the DAG run actually 'ran.' Aside from the DAG +run's start and end date, there is another date called ``logical date``, which +describes the intended time for the DAG run to be scheduled or triggered to +run. The reason why this is called 'logical' is because of the abstract nature +of it having multiple meanings, depending on the context of the DAG run itself. + +For example, when the DAG run is being triggered, the logical date would be the +date and time of which the DAG run was triggered, and the value should be equal +to DAG run's start date. However, when the DAG is being scheduled, with certain +schedule interval put in place, this logical date is going to indicate the time +at which it marks the start of the data interval, where the DAG run's start +date would then be the logical date + scheduled interval. DAG Assignment -------------- From e126e2ef8cf7f978f8000eefaa8d89ff5f9ab8eb Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Tue, 8 Feb 2022 15:01:54 -0600 Subject: [PATCH 02/11] Update dags.rst --- docs/apache-airflow/concepts/dags.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 9cae2fbd09bca..91bbd476f49d2 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -179,10 +179,11 @@ Tasks specified inside a DAG are also instantiated into When the DAG run starts, it will have the start date and end date. This period describes the time when the DAG run actually 'ran.' Aside from the DAG -run's start and end date, there is another date called ``logical date``, which -describes the intended time for the DAG run to be scheduled or triggered to -run. The reason why this is called 'logical' is because of the abstract nature -of it having multiple meanings, depending on the context of the DAG run itself. +run's start and end date, there is another date called ``logical date`` (formally +known as execution date), which describes the intended time for the DAG run to +be scheduled or triggered to run. The reason why this is called 'logical' is +because of the abstract nature of it having multiple meanings, depending on the +context of the DAG run itself. For example, when the DAG run is being triggered, the logical date would be the date and time of which the DAG run was triggered, and the value should be equal From d31ec187d992973bf6c344185d058635e5f053d2 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Thu, 10 Feb 2022 09:31:35 -0600 Subject: [PATCH 03/11] Update dags.rst changed a bit of formatting --- docs/apache-airflow/concepts/dags.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 91bbd476f49d2..176cd2ab4c6ea 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -179,11 +179,11 @@ Tasks specified inside a DAG are also instantiated into When the DAG run starts, it will have the start date and end date. This period describes the time when the DAG run actually 'ran.' Aside from the DAG -run's start and end date, there is another date called ``logical date`` (formally -known as execution date), which describes the intended time for the DAG run to -be scheduled or triggered to run. The reason why this is called 'logical' is -because of the abstract nature of it having multiple meanings, depending on the -context of the DAG run itself. +run's start and end date, there is another date called ``logical date`` +(formally known as execution date), which describes the intended time for the +DAG run to be scheduled or triggered to run. The reason why this is called +``logical`` is because of the abstract nature of it having multiple meanings, +depending on the context of the DAG run itself. For example, when the DAG run is being triggered, the logical date would be the date and time of which the DAG run was triggered, and the value should be equal From dbb016c6f21b2197b029379d3d3ce2b555d94d0b Mon Sep 17 00:00:00 2001 From: howardyoo Date: Wed, 16 Feb 2022 01:19:32 -0600 Subject: [PATCH 04/11] added 'see also' section on three different areas of documentation to interlink each other --- docs/apache-airflow/concepts/dags.rst | 2 ++ docs/apache-airflow/dag-run.rst | 2 ++ docs/apache-airflow/faq.rst | 7 +++++++ 3 files changed, 11 insertions(+) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 176cd2ab4c6ea..96a8e68218b27 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -157,6 +157,8 @@ The ``schedule_interval`` argument takes any value that is a valid `Crontab `. If ``schedule_interval`` is not enough to express the DAG's schedule, see :doc:`Timetables `. + For more information on ``logical date``, see :ref:`data-interval` and + :ref:`faq:what-does-execution-date-mean`. Every time you run a DAG, you are creating a new instance of that DAG which Airflow calls a :doc:`DAG Run `. DAG Runs can run in parallel for the diff --git a/docs/apache-airflow/dag-run.rst b/docs/apache-airflow/dag-run.rst index 90bb404696106..d0c7a63f8742c 100644 --- a/docs/apache-airflow/dag-run.rst +++ b/docs/apache-airflow/dag-run.rst @@ -84,6 +84,8 @@ scheduled one interval after ``start_date``. If ``schedule_interval`` is not enough to express your DAG's schedule, logical date, or data interval, see :doc:`/concepts/timetable`. + For more information on ``logical date``, see :ref:`concepts:dag-run` and + :ref:`faq:what-does-execution-date-mean` Re-run DAG '''''''''' diff --git a/docs/apache-airflow/faq.rst b/docs/apache-airflow/faq.rst index 857e685abbda2..1120c250db917 100644 --- a/docs/apache-airflow/faq.rst +++ b/docs/apache-airflow/faq.rst @@ -214,6 +214,8 @@ This allows for a backfill on tasks that have ``depends_on_past=True`` to actually start. If this were not the case, the backfill just would not start. +.. _faq:what-does-execution-date-mean: + What does ``execution_date`` mean? ---------------------------------- @@ -248,6 +250,11 @@ misunderstandings. Note that ``ds`` (the YYYY-MM-DD form of ``data_interval_start``) refers to *date* ***string***, not *date* ***start*** as may be confusing to some. +.. tip:: + + For more information on ``logical date``, see :ref:`data-interval` and + :ref:`concepts:dag-run:`. + How to create DAGs dynamically? ------------------------------- From 9b923743ecc8aa568b4d4e7bc72e8dcf1936abf7 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 01:44:26 -0600 Subject: [PATCH 05/11] Update dags.rst --- docs/apache-airflow/concepts/dags.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 96a8e68218b27..614ffcba280f1 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -157,7 +157,7 @@ The ``schedule_interval`` argument takes any value that is a valid `Crontab `. If ``schedule_interval`` is not enough to express the DAG's schedule, see :doc:`Timetables `. - For more information on ``logical date``, see :ref:`data-interval` and + For more information on ``logical date``, see :ref:`data-interval` and :ref:`faq:what-does-execution-date-mean`. Every time you run a DAG, you are creating a new instance of that DAG which @@ -180,18 +180,18 @@ Tasks specified inside a DAG are also instantiated into :ref:`Task Instances ` along with it. When the DAG run starts, it will have the start date and end date. This period -describes the time when the DAG run actually 'ran.' Aside from the DAG -run's start and end date, there is another date called ``logical date`` -(formally known as execution date), which describes the intended time for the -DAG run to be scheduled or triggered to run. The reason why this is called -``logical`` is because of the abstract nature of it having multiple meanings, +describes the time when the DAG run actually 'ran.' Aside from the DAG +run's start and end date, there is another date called ``logical date`` +(formally known as execution date), which describes the intended time for the +DAG run to be scheduled or triggered to run. The reason why this is called +``logical`` is because of the abstract nature of it having multiple meanings, depending on the context of the DAG run itself. For example, when the DAG run is being triggered, the logical date would be the date and time of which the DAG run was triggered, and the value should be equal to DAG run's start date. However, when the DAG is being scheduled, with certain schedule interval put in place, this logical date is going to indicate the time -at which it marks the start of the data interval, where the DAG run's start +at which it marks the start of the data interval, where the DAG run's start date would then be the logical date + scheduled interval. DAG Assignment From 57e3074f3c80415cced339a982d069457701ebc3 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 01:45:19 -0600 Subject: [PATCH 06/11] Update dag-run.rst --- docs/apache-airflow/dag-run.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/apache-airflow/dag-run.rst b/docs/apache-airflow/dag-run.rst index d0c7a63f8742c..62555b10ed3ab 100644 --- a/docs/apache-airflow/dag-run.rst +++ b/docs/apache-airflow/dag-run.rst @@ -84,7 +84,7 @@ scheduled one interval after ``start_date``. If ``schedule_interval`` is not enough to express your DAG's schedule, logical date, or data interval, see :doc:`/concepts/timetable`. - For more information on ``logical date``, see :ref:`concepts:dag-run` and + For more information on ``logical date``, see :ref:`concepts:dag-run` and :ref:`faq:what-does-execution-date-mean` Re-run DAG From 801e9b6d8b01578f81a6ef62fc84c7c9908a9fe7 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 01:45:51 -0600 Subject: [PATCH 07/11] Update faq.rst --- docs/apache-airflow/faq.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/apache-airflow/faq.rst b/docs/apache-airflow/faq.rst index 1120c250db917..0d6b96b38021f 100644 --- a/docs/apache-airflow/faq.rst +++ b/docs/apache-airflow/faq.rst @@ -252,7 +252,7 @@ Note that ``ds`` (the YYYY-MM-DD form of ``data_interval_start``) refers to .. tip:: - For more information on ``logical date``, see :ref:`data-interval` and + For more information on ``logical date``, see :ref:`data-interval` and :ref:`concepts:dag-run:`. From 14767b2b419579d3cc03da6b137dc90c39262e95 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 10:21:28 -0600 Subject: [PATCH 08/11] Update faq.rst fixed the incorrect typo --- docs/apache-airflow/faq.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/apache-airflow/faq.rst b/docs/apache-airflow/faq.rst index 0d6b96b38021f..7f72a0d162f5f 100644 --- a/docs/apache-airflow/faq.rst +++ b/docs/apache-airflow/faq.rst @@ -253,7 +253,7 @@ Note that ``ds`` (the YYYY-MM-DD form of ``data_interval_start``) refers to .. tip:: For more information on ``logical date``, see :ref:`data-interval` and - :ref:`concepts:dag-run:`. + :ref:`concepts:dag-run`. How to create DAGs dynamically? From 317fe4b35c229cfe0b94817d47882425d58229d2 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 23:05:48 -0600 Subject: [PATCH 09/11] Update docs/apache-airflow/concepts/dags.rst sounds good! I'll commit the changes. Co-authored-by: Tzu-ping Chung --- docs/apache-airflow/concepts/dags.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 614ffcba280f1..3e18e1fa5a22f 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -181,10 +181,10 @@ Tasks specified inside a DAG are also instantiated into When the DAG run starts, it will have the start date and end date. This period describes the time when the DAG run actually 'ran.' Aside from the DAG -run's start and end date, there is another date called ``logical date`` -(formally known as execution date), which describes the intended time for the -DAG run to be scheduled or triggered to run. The reason why this is called -``logical`` is because of the abstract nature of it having multiple meanings, +run's start and end date, there is another date called *logical date* +(formally known as execution date), which describes the intended time a +DAG run is scheduled or triggered. The reason why this is called +*logical* is because of the abstract nature of it having multiple meanings, depending on the context of the DAG run itself. For example, when the DAG run is being triggered, the logical date would be the From 3479231f94840e8bd602f8a1eb4912d357c3b117 Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 23:10:33 -0600 Subject: [PATCH 10/11] Update dags.rst dag run does not have end date when it starts, but when it ends. Modified the description correctly. --- docs/apache-airflow/concepts/dags.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 3e18e1fa5a22f..8afce9ddc20c6 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -179,8 +179,8 @@ In much the same way a DAG instantiates into a DAG Run every time it's run, Tasks specified inside a DAG are also instantiated into :ref:`Task Instances ` along with it. -When the DAG run starts, it will have the start date and end date. This period -describes the time when the DAG run actually 'ran.' Aside from the DAG +A DAG run will have a start date when it starts, and end date when it ends. +This period describes the time when the DAG actually 'ran.' Aside from the DAG run's start and end date, there is another date called *logical date* (formally known as execution date), which describes the intended time a DAG run is scheduled or triggered. The reason why this is called From baad2e376523240b10c2fffd824b0aa8fb1dbffc Mon Sep 17 00:00:00 2001 From: Howard Yoo <32691630+howardyoo@users.noreply.github.com> Date: Wed, 16 Feb 2022 23:11:33 -0600 Subject: [PATCH 11/11] Update docs/apache-airflow/concepts/dags.rst Co-authored-by: Tzu-ping Chung --- docs/apache-airflow/concepts/dags.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/apache-airflow/concepts/dags.rst b/docs/apache-airflow/concepts/dags.rst index 8afce9ddc20c6..e339abeda65bc 100644 --- a/docs/apache-airflow/concepts/dags.rst +++ b/docs/apache-airflow/concepts/dags.rst @@ -187,10 +187,10 @@ DAG run is scheduled or triggered. The reason why this is called *logical* is because of the abstract nature of it having multiple meanings, depending on the context of the DAG run itself. -For example, when the DAG run is being triggered, the logical date would be the +For example, if a DAG run is manually triggered by the user, its logical date would be the date and time of which the DAG run was triggered, and the value should be equal -to DAG run's start date. However, when the DAG is being scheduled, with certain -schedule interval put in place, this logical date is going to indicate the time +to DAG run's start date. However, when the DAG is being automatically scheduled, with certain +schedule interval put in place, the logical date is going to indicate the time at which it marks the start of the data interval, where the DAG run's start date would then be the logical date + scheduled interval.