Adding extra requirements for build and runtime of the PROD image. (#…

…16170)

This PR adds capability of adding extra requirements to PROD image:

1) During the build by placing requirements.txt in the
   ``docker-context-files`` folder

2) During execution of the container - by passing
   _PIP_ADDITIONAL_REQUIREMENTS variable

The second case is only useful durint quick test/development and
should not be used in production.

Also updated documentation to contain all development/test
variables for docker compose and clarifying that the options
starting with _ are ment to be only used for quick testing.

Dockerfile

@@ -264,6 +264,13 @@ RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \
     find /root/.local -executable -print0 | xargs --null chmod g+x; \
     find /root/.local -print0 | xargs --null chmod g+rw
 
+# In case there is a requirements.txt file in "docker-context-files" it will be installed
+# during the build additionally to whatever has been installed so far. It is recommended that
+# the requirements.txt contains only dependencies with == version specification
+RUN if [[ -f /docker-context-files/requirements.txt ]]; then \
+        pip install --no-cache-dir --user -r /docker-context-files/requirements.txt; \
+    fi
+
 ARG BUILD_ID
 ARG COMMIT_SHA
 ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"

IMAGES.rst

@@ -495,7 +495,7 @@ additional apt dev and runtime dependencies.
     --build-arg ADDITIONAL_PYTHON_DEPS="pandas"
     --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++"
     --build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless"
-    --tag my-image
+    --tag my-image:0.0.1
 
 
 the same image can be built using ``breeze`` (it supports auto-completion of the options):
@@ -533,7 +533,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
     --build-arg ADDITIONAL_RUNTIME_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add --no-tty - && curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
     --build-arg ADDITIONAL_RUNTIME_APT_DEPS="msodbcsql17 unixodbc git procps vim" \
     --build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
-    --tag my-image
+    --tag my-image:0.0.1
 
 CI image build arguments
 ------------------------
@@ -664,44 +664,45 @@ This builds the CI image in version 3.7 with default extras ("all").
 
 .. code-block:: bash
 
-  docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster"
+  docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --tag my-image:0.0.1
 
 
 This builds the CI image in version 3.6 with "gcp" extra only.
 
 .. code-block:: bash
 
   docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
-    --build-arg AIRFLOW_EXTRAS=gcp
+    --build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1
 
 
 This builds the CI image in version 3.6 with "apache-beam" extra added.
 
 .. code-block:: bash
 
   docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
-    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam"
+    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1
 
 This builds the CI image in version 3.6 with "mssql" additional package added.
 
 .. code-block:: bash
 
   docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
-    --build-arg ADDITIONAL_PYTHON_DEPS="mssql"
+    --build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1
 
 This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev dependencies added.
 
 .. code-block::
 
   docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
-    --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++"
+    --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1
 
 This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added.
 
 .. code-block::
 
   docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
-    --build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless"
+    --build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" \
+    --tag my-image:0.0.1
 
 CI Image manifests
 ------------------

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -601,14 +601,14 @@ USER ${AIRFLOW_UID}
 To build an image build and run a shell, run:
 
 ```shell script
-docker build . -t my-airflow
+docker build . --tag my-image:0.0.1
 docker run  -ti \
     --rm \
     -v "$PWD/data:/opt/airflow/" \
     -v "$PWD/keys/:/keys/" \
     -p 8080:8080 \
     -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
-    my-airflow bash
+    my-image:0.0.1 bash
 ```
 
 ### Additional Verification

dev/check_files.py

@@ -47,7 +47,7 @@
 
 
 DOCKER_CMD = """
-docker build -t local/airflow .
+docker build --tag local/airflow .
 docker local/airflow info
 """
 
@@ -80,7 +80,7 @@ def create_docker(txt: str):
     print("\n[bold]To check installation run:[/bold]")
     print(
         """\
-        docker build -f Dockerfile.pmc -t local/airflow .
+        docker build -f Dockerfile.pmc --tag local/airflow .
         docker run local/airflow info
         """
     )

docs/apache-airflow/index.rst

@@ -101,7 +101,6 @@ unit of work and continuity.
     changelog
     best-practices
     production-deployment
-    backport-providers
     faq
     privacy_notice
 

docs/apache-airflow/start/docker-compose.yaml

@@ -23,16 +23,21 @@
 # This configuration supports basic configuration using environment variables or an .env file
 # The following variables are supported:
 #
-# AIRFLOW_IMAGE_NAME         - Docker image name used to run Airflow.
-#                              Default: apache/airflow:|version|
-# AIRFLOW_UID                - User ID in Airflow containers
-#                              Default: 50000
-# AIRFLOW_GID                - Group ID in Airflow containers
-#                              Default: 50000
-# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
-#                              Default: airflow
-# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
-#                              Default: airflow
+# AIRFLOW_IMAGE_NAME           - Docker image name used to run Airflow.
+#                                Default: apache/airflow:|version|
+# AIRFLOW_UID                  - User ID in Airflow containers
+#                                Default: 50000
+# AIRFLOW_GID                  - Group ID in Airflow containers
+#                                Default: 50000
+#
+# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
+#
+# _AIRFLOW_WWW_USER_USERNAME   - Username for the administrator account (if requested).
+#                                Default: airflow
+# _AIRFLOW_WWW_USER_PASSWORD   - Password for the administrator account (if requested).
+#                                Default: airflow
+# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
+#                                Default: ''
 #
 # Feel free to modify this file to suit your needs.
 ---
@@ -50,6 +55,7 @@ x-airflow-common:
     AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
     AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
     AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
+    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
   volumes:
     - ./dags:/opt/airflow/dags
     - ./logs:/opt/airflow/logs

docs/apache-airflow/start/docker.rst

@@ -15,6 +15,8 @@
     specific language governing permissions and limitations
     under the License.
 
+.. _running-airflow-in-docker:
+
 Running Airflow in Docker
 #########################
 
@@ -73,7 +75,8 @@ Some directories in the container are mounted, which means that their contents a
 - ``./logs`` - contains logs from task execution and scheduler.
 - ``./plugins`` - you can put your :doc:`custom plugins </plugins>` here.
 
-This file uses the latest Airflow image (`apache/airflow <https://hub.docker.com/r/apache/airflow>`__). If you need install a new Python library or system library, you can :doc:`customize and extend it <docker-stack:index>`.
+This file uses the latest Airflow image (`apache/airflow <https://hub.docker.com/r/apache/airflow>`__).
+If you need install a new Python library or system library, you can :doc:`build your image <docker-stack:index>`.
 
 .. _initializing_docker_compose_environment:
 
@@ -260,13 +263,26 @@ runtime user id which is unknown at the time of building the image.
 |                                | you want to use different UID than default it must  |                          |
 |                                | be set to ``0``.                                    |                          |
 +--------------------------------+-----------------------------------------------------+--------------------------+
-| ``_AIRFLOW_WWW_USER_USERNAME`` | Username for the administrator UI account.          |                          |
-|                                | If this value is specified, admin UI user gets      |                          |
-|                                | created automatically. This is only useful when     |                          |
-|                                | you want to run Airflow for a test-drive and        |                          |
-|                                | want to start a container with embedded development |                          |
-|                                | database.                                           |                          |
-+--------------------------------+-----------------------------------------------------+--------------------------+
-| ``_AIRFLOW_WWW_USER_PASSWORD`` | Password for the administrator UI account.          |                          |
-|                                | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set.  |                          |
-+--------------------------------+-----------------------------------------------------+--------------------------+
+
+Those additional variables are useful in case you are trying out/testing Airflow installation via docker compose.
+They are not intended to be used in production, but they make the environment faster to bootstrap for first time
+users with the most common customizations.
+
++----------------------------------+-----------------------------------------------------+--------------------------+
+|   Variable                       | Description                                         | Default                  |
++==================================+=====================================================+==========================+
+| ``_AIRFLOW_WWW_USER_USERNAME``   | Username for the administrator UI account.          | airflow                  |
+|                                  | If this value is specified, admin UI user gets      |                          |
+|                                  | created automatically. This is only useful when     |                          |
+|                                  | you want to run Airflow for a test-drive and        |                          |
+|                                  | want to start a container with embedded development |                          |
+|                                  | database.                                           |                          |
++----------------------------------+-----------------------------------------------------+--------------------------+
+| ``_AIRFLOW_WWW_USER_PASSWORD``   | Password for the administrator UI account.          | airflow                  |
+|                                  | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set.  |                          |
++----------------------------------+-----------------------------------------------------+--------------------------+
+| ``_PIP_ADDITIONAL_REQUIREMENTS`` | If not empty, airflow containers will attempt to    |                          |
+|                                  | install requirements specified in the variable.     |                          |
+|                                  | example: ``lxml==4.6.3 charset-normalizer==1.4.1``. |                          |
+|                                  | Available in Airflow image 2.1.1 and above.         |                          |
++----------------------------------+-----------------------------------------------------+--------------------------+

docs/conf.py

@@ -258,10 +258,15 @@ def _get_rst_filepath_from_path(filepath: str):
     html_extra_with_substitutions = [
         f"{ROOT_DIR}/docs/apache-airflow/start/docker-compose.yaml",
     ]
+    # Replace "|version|" in links
     manual_substitutions_in_generated_html = [
         "installation.html",
     ]
 
+if PACKAGE_NAME == 'docker-stack':
+    # Replace "|version|" inside ```` quotes
+    manual_substitutions_in_generated_html = ["build.html"]
+
 # -- Theme configuration -------------------------------------------------------
 # Custom sidebar templates, maps document names to template names.
 html_sidebars = {