Skip to content

Commit

Permalink
Doc: Recommend using same configs on all Airflow components (#17146)
Browse files Browse the repository at this point in the history
Some users seem to be using different configs for different components. This doc makes it clear on what we recommend.

Context: apache/airflow#16754 (comment)

>I'm new to Airflow and I haven't seen an explicit part within the docs that says "all configuration must be shared between airflow components". In fact, all configuration keys are prefixed with some naming scheme which led me to think that some are common, some are for webserver etc... I personally don't like to have common configuration because changing a single setting requires a change in all components (hence restart) but I get that this is intentional for Airflow. I think this can be better communicated throughout the docs.

GitOrigin-RevId: 3a263ab5b0b8b8b796d96f060e6ccd4a0e379bf3
  • Loading branch information
kaxil authored and Cloud Composer Team committed Jan 27, 2023
1 parent 64b23ec commit ee1b7c4
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 0 deletions.
5 changes: 5 additions & 0 deletions docs/apache-airflow/configurations-ref.rst
Expand Up @@ -22,6 +22,11 @@ Configuration Reference
This page contains the list of all the available Airflow configurations that you
can set in ``airflow.cfg`` file or using environment variables.

Use the same configuration across all the Airflow components. While each component
does not require all, some configurations need to be same otherwise they would not
work as expected. A good example for that is :ref:`secret_key<config:webserver__secret_key>` which
should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.

.. note::
For more information on setting the configuration, see :doc:`howto/set-config`

Expand Down
6 changes: 6 additions & 0 deletions docs/apache-airflow/howto/set-config.rst
Expand Up @@ -115,3 +115,9 @@ the example below.

.. note::
See :doc:`../modules_management` for details on how Python and Airflow manage modules.

.. note::
Use the same configuration across all the Airflow components. While each component
does not require all, some configurations need to be same otherwise they would not
work as expected. A good example for that is :ref:`secret_key<config:webserver__secret_key>` which
should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.

0 comments on commit ee1b7c4

Please sign in to comment.