From 6395dbe0c3a1871c5ff46f0805e5c920391dda22 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Wed, 2 Jun 2021 01:08:44 +0200 Subject: [PATCH] Adding extra requirements for build and runtime of the PROD image. (#16170) This PR adds capability of adding extra requirements to PROD image: 1) During the build by placing requirements.txt in the ``docker-context-files`` folder 2) During execution of the container - by passing _PIP_ADDITIONAL_REQUIREMENTS variable The second case is only useful durint quick test/development and should not be used in production. Also updated documentation to contain all development/test variables for docker compose and clarifying that the options starting with _ are ment to be only used for quick testing. (cherry picked from commit d245992d2ac0d781b6b55fd030a076ca5c799bf7) --- Dockerfile | 7 + IMAGES.rst | 17 +- dev/README_RELEASE_PROVIDER_PACKAGES.md | 4 +- dev/check_files.py | 4 +- docs/apache-airflow/index.rst | 1 - docs/apache-airflow/start/docker-compose.yaml | 26 ++- docs/apache-airflow/start/docker.rst | 38 +++- docs/conf.py | 5 + docs/docker-stack/build.rst | 213 ++++++++++++++---- .../customizing/add-build-essential-custom.sh | 4 +- .../customizing/custom-sources.sh | 4 +- .../github-different-repository.sh | 4 +- .../{github-master.sh => github-main.sh} | 4 +- .../customizing/github-v2-1-test.sh | 4 +- .../customizing/pypi-dev-runtime-deps.sh | 4 +- .../customizing/pypi-extras-and-deps.sh | 4 +- .../customizing/pypi-selected-version.sh | 4 +- .../customizing/stable-airflow.sh | 4 +- .../extending/add-apt-packages/Dockerfile | 2 +- .../add-build-essential-extend/Dockerfile | 2 +- .../extending/add-pypi-packages/Dockerfile | 2 +- .../extending/embedding-dags/Dockerfile | 2 +- .../extending/writable-directory/Dockerfile | 2 +- .../restricted/restricted_environments.sh | 3 +- docs/docker-stack/entrypoint.rst | 150 +++++++----- docs/docker-stack/index.rst | 31 ++- docs/docker-stack/recipes.rst | 4 +- docs/helm-chart/production-guide.rst | 27 ++- docs/helm-chart/quick-start.rst | 102 ++++++++- scripts/in_container/prod/entrypoint_prod.sh | 16 ++ 30 files changed, 519 insertions(+), 175 deletions(-) rename docs/docker-stack/docker-examples/customizing/{github-master.sh => github-main.sh} (94%) diff --git a/Dockerfile b/Dockerfile index 0d872acd7003f..444f0a549ac2b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -267,6 +267,13 @@ RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \ find /root/.local -executable -print0 | xargs --null chmod g+x; \ find /root/.local -print0 | xargs --null chmod g+rw +# In case there is a requirements.txt file in "docker-context-files" it will be installed +# during the build additionally to whatever has been installed so far. It is recommended that +# the requirements.txt contains only dependencies with == version specification +RUN if [[ -f /docker-context-files/requirements.txt ]]; then \ + pip install --no-cache-dir --user -r /docker-context-files/requirements.txt; \ + fi + ARG BUILD_ID ARG COMMIT_SHA ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow" diff --git a/IMAGES.rst b/IMAGES.rst index 6b4ca3117db92..d332b99b528bf 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -495,7 +495,7 @@ additional apt dev and runtime dependencies. --build-arg ADDITIONAL_PYTHON_DEPS="pandas" --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless" - --tag my-image + --tag my-image:0.0.1 the same image can be built using ``breeze`` (it supports auto-completion of the options): @@ -533,7 +533,7 @@ based on example in `this comment ` here. -This file uses the latest Airflow image (`apache/airflow `__). If you need install a new Python library or system library, you can :doc:`customize and extend it `. +This file uses the latest Airflow image (`apache/airflow `__). +If you need install a new Python library or system library, you can :doc:`build your image `. .. _initializing_docker_compose_environment: @@ -247,13 +250,26 @@ runtime user id which is unknown at the time of building the image. | | you want to use different UID than default it must | | | | be set to ``0``. | | +--------------------------------+-----------------------------------------------------+--------------------------+ -| ``_AIRFLOW_WWW_USER_USERNAME`` | Username for the administrator UI account. | | -| | If this value is specified, admin UI user gets | | -| | created automatically. This is only useful when | | -| | you want to run Airflow for a test-drive and | | -| | want to start a container with embedded development | | -| | database. | | -+--------------------------------+-----------------------------------------------------+--------------------------+ -| ``_AIRFLOW_WWW_USER_PASSWORD`` | Password for the administrator UI account. | | -| | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set. | | -+--------------------------------+-----------------------------------------------------+--------------------------+ + +Those additional variables are useful in case you are trying out/testing Airflow installation via docker compose. +They are not intended to be used in production, but they make the environment faster to bootstrap for first time +users with the most common customizations. + ++----------------------------------+-----------------------------------------------------+--------------------------+ +| Variable | Description | Default | ++==================================+=====================================================+==========================+ +| ``_AIRFLOW_WWW_USER_USERNAME`` | Username for the administrator UI account. | airflow | +| | If this value is specified, admin UI user gets | | +| | created automatically. This is only useful when | | +| | you want to run Airflow for a test-drive and | | +| | want to start a container with embedded development | | +| | database. | | ++----------------------------------+-----------------------------------------------------+--------------------------+ +| ``_AIRFLOW_WWW_USER_PASSWORD`` | Password for the administrator UI account. | airflow | +| | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set. | | ++----------------------------------+-----------------------------------------------------+--------------------------+ +| ``_PIP_ADDITIONAL_REQUIREMENTS`` | If not empty, airflow containers will attempt to | | +| | install requirements specified in the variable. | | +| | example: ``lxml==4.6.3 charset-normalizer==1.4.1``. | | +| | Available in Airflow image 2.1.1 and above. | | ++----------------------------------+-----------------------------------------------------+--------------------------+ diff --git a/docs/conf.py b/docs/conf.py index e403ffeae4ee9..1e798dfa3a6c2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -258,10 +258,15 @@ def _get_rst_filepath_from_path(filepath: str): html_extra_with_substitutions = [ f"{ROOT_DIR}/docs/apache-airflow/start/docker-compose.yaml", ] + # Replace "|version|" in links manual_substitutions_in_generated_html = [ "installation.html", ] +if PACKAGE_NAME == 'docker-stack': + # Replace "|version|" inside ```` quotes + manual_substitutions_in_generated_html = ["build.html"] + # -- Theme configuration ------------------------------------------------------- # Custom sidebar templates, maps document names to template names. html_sidebars = { diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst index 3a5a977cfa758..46f87fb4acfc3 100644 --- a/docs/docker-stack/build.rst +++ b/docs/docker-stack/build.rst @@ -15,16 +15,134 @@ specific language governing permissions and limitations under the License. +.. _build:build_image: + Building the image ================== -Before you dive-deeply in the way how the Airflow Image is build, named and why we are doing it the -way we do, you might want to know very quickly how you can extend or customize the existing image -for Apache Airflow. This chapter gives you a short answer to those questions. +Before you dive-deeply in the way how the Airflow Image is build, let us first explain why you might need +to build the custom container image and we show a few typical ways you can do it. + +Why custom image ? +------------------ + +The Apache Airflow community, releases Docker Images which are ``reference images`` for Apache Airflow. +However, Airflow has more than 60 community managed providers (installable via extras) and some of the +default extras/providers installed are not used by everyone, sometimes others extras/providers +are needed, sometimes (very often actually) you need to add your own custom dependencies, +packages or even custom providers. + +In Kubernetes and Docker terms this means that you need another image with your specific requirements. +This is why you should learn how to build your own Docker (or more properly Container) image. +You might be tempted to use the ``reference image`` and dynamically install the new packages while +starting your containers, but this is a bad idea for multiple reasons - starting from fragility of the build +and ending with the extra time needed to install those packages - which has to happen every time every +container starts. The only viable way to deal with new dependencies and requirements in production is to +build and use your own image. You should only use installing dependencies dynamically in case of +"hobbyist" and "quick start" scenarios when you want to iterate quickly to try things out and later +replace it with your own images. + +How to build your own image +--------------------------- + +There are several most-typical scenarios that you will encounter and here is a quick recipe on how to achieve +your goal quickly. In order to understand details you can read further, but for the simple cases using +typical tools here are the simple examples. + +In the simplest case building your image consists of those steps: + +1) Create your own ``Dockerfile`` (name it ``Dockerfile``) where you add: + +* information what your image should be based on (for example ``FROM: apache/airflow:|version|-python3.8`` + +* additional steps that should be executed in your image (typically in the form of ``RUN ``) + +2) Build your image. This can be done with ``docker`` CLI tools and examples below assume ``docker`` is used. + There are other tools like ``kaniko`` or ``podman`` that allow you to build the image, but ``docker`` is + so far the most popular and developer-friendly tool out there. Typical way of building the image looks + like follows (``my-image:0.0.1`` is the custom tag of your image containing version). + In case you use some kind of registry where you will be using the image from, it is usually named + in the form of ``registry/image-name``. The name of the image has to be configured for the deployment + method your image will be deployed. This can be set for example as image name in the + `docker-compose file `_ or in the `Helm chart `_. + +.. code-block:: shell + + docker build . -f Dockerfile --tag my-image:0.0.1 + +3) [Optional] Test the image. Airflow contains tool that allows you to test the image. This step however, + requires locally checked out or extracted Airflow sources. If you happen to have the sources you can + test the image by running this command (in airflow root folder). The output will tell you if the image + is "good-to-go". + +.. code-block:: shell + + ./scripts/ci/tools/verify_docker_image.sh PROD my-image:0.0.1 + +4) Once you build the image locally you have usually several options to make them available for your deployment: + +* For ``docker-compose`` deployment, that's all you need. The image is stored in docker engine cache + and docker compose will use it from there. + +* For some - development targeted - Kubernetes deployments you can load the images directly to + Kubernetes clusters. Clusters such as ``kind`` or ``minikube`` have dedicated ``load`` method to load the + images to the cluster. + +* Last but not least - you can push your image to a remote registry which is the most common way + of storing and exposing the images, and it is most portable way of publishing the image. Both + Docker-Compose and Kubernetes can make use of images exposed via registries. + +The most common scenarios where you want to build your own image are adding a new ``apt`` package, +adding a new ``PyPI`` dependency and embedding DAGs into the image. +Example Dockerfiles for those scenarios are below, and you can read further +for more complex cases which might involve either extending or customizing the image. + +Adding new ``apt`` package +.......................... + +The following example adds ``vim`` to the airflow image. + +.. exampleinclude:: docker-examples/extending/add-apt-packages/Dockerfile + :language: Dockerfile + :start-after: [START Dockerfile] + :end-before: [END Dockerfile] + + +Adding a new ``PyPI`` package +............................. + +The following example adds ``lxml`` python package from PyPI to the image. + +.. exampleinclude:: docker-examples/extending/add-pypi-packages/Dockerfile + :language: Dockerfile + :start-after: [START Dockerfile] + :end-before: [END Dockerfile] + +Embedding DAGs +.............. + +The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/dags`` folder. + +.. exampleinclude:: docker-examples/extending/embedding-dags/Dockerfile + :language: Dockerfile + :start-after: [START Dockerfile] + :end-before: [END Dockerfile] + + +.. exampleinclude:: docker-examples/extending/embedding-dags/test_dag.py + :language: Python + :start-after: [START dag] + :end-before: [END dag] + + Extending vs. customizing the image ----------------------------------- +You might want to know very quickly how you can extend or customize the existing image +for Apache Airflow. This chapter gives you a short answer to those questions. + + Here is the comparison of the two types of building images. Here is your guide if you want to choose how you want to build your image. @@ -132,8 +250,8 @@ You should be aware, about a few things: Examples of image extending --------------------------- -An ``apt`` package example -.......................... +Example of adding ``apt`` package +................................. The following example adds ``vim`` to the airflow image. @@ -142,8 +260,8 @@ The following example adds ``vim`` to the airflow image. :start-after: [START Dockerfile] :end-before: [END Dockerfile] -A ``PyPI`` package example -.......................... +Example of adding ``PyPI`` package +.................................. The following example adds ``lxml`` python package from PyPI to the image. @@ -152,8 +270,8 @@ The following example adds ``lxml`` python package from PyPI to the image. :start-after: [START Dockerfile] :end-before: [END Dockerfile] -A ``umask`` requiring example -............................. +Example when writable directory is needed +......................................... The following example adds a new directory that is supposed to be writable for any arbitrary user running the container. @@ -164,8 +282,8 @@ running the container. :end-before: [END Dockerfile] -A ``build-essential`` requiring package example -............................................... +Example when you add packages requiring compilation +................................................... The following example adds ``mpi4py`` package which requires both ``build-essential`` and ``mpi compiler``. @@ -177,8 +295,8 @@ The following example adds ``mpi4py`` package which requires both ``build-essent The size of this image is ~ 1.1 GB when build. As you will see further, you can achieve 20% reduction in size of the image in case you use "Customizing" rather than "Extending" the image. -DAG embedding example -..................... +Example when you want to embed DAGs +................................... The following example adds ``test_dag.py`` to your image in the ``/opt/airflow/dags`` folder. @@ -223,27 +341,28 @@ to add extra dependencies needed at early stages of image building. When customizing the image you can choose a number of options how you install Airflow: - * From the PyPI releases (default) - * From the custom installation sources - using additional/replacing the original apt or PyPI repositories - * From local sources. This is used mostly during development. - * From tag or branch, or specific commit from a GitHub Airflow repository (or fork). This is particularly - useful when you build image for a custom version of Airflow that you keep in your fork and you do not - want to release the custom Airflow version to PyPI. - * From locally stored binary packages for Airflow, Airflow Providers and other dependencies. This is - particularly useful if you want to build Airflow in a highly-secure environment where all such packages - must be vetted by your security team and stored in your private artifact registry. This also - allows to build airflow image in an air-gaped environment. - * Side note. Building ``Airflow`` in an ``air-gaped`` environment sounds pretty funny, doesn't it? +* From the PyPI releases (default) +* From the custom installation sources - using additional/replacing the original apt or PyPI repositories +* From local sources. This is used mostly during development. +* From tag or branch, or specific commit from a GitHub Airflow repository (or fork). This is particularly + useful when you build image for a custom version of Airflow that you keep in your fork and you do not + want to release the custom Airflow version to PyPI. +* From locally stored binary packages for Airflow, Airflow Providers and other dependencies. This is + particularly useful if you want to build Airflow in a highly-secure environment where all such packages + must be vetted by your security team and stored in your private artifact registry. This also + allows to build airflow image in an air-gaped environment. +* Side note. Building ``Airflow`` in an ``air-gaped`` environment sounds pretty funny, doesn't it? You can also add a range of customizations while building the image: - * base python image you use for Airflow - * version of Airflow to install - * extras to install for Airflow (or even removing some default extras) - * additional apt/python dependencies to use while building Airflow (DEV dependencies) - * additional apt/python dependencies to install for runtime version of Airflow (RUNTIME dependencies) - * additional commands and variables to set if needed during building or preparing Airflow runtime - * choosing constraint file to use when installing Airflow +* base python image you use for Airflow +* version of Airflow to install +* extras to install for Airflow (or even removing some default extras) +* additional apt/python dependencies to use while building Airflow (DEV dependencies) +* add ``requirements.txt`` file to ``docker-context-files`` directory to add extra requirements +* additional apt/python dependencies to install for runtime version of Airflow (RUNTIME dependencies) +* additional commands and variables to set if needed during building or preparing Airflow runtime +* choosing constraint file to use when installing Airflow Additional explanation is needed for the last point. Airflow uses constraints to make sure that it can be predictably installed, even if some new versions of Airflow dependencies are @@ -262,6 +381,12 @@ of constraints that you manually prepared. You can read more about constraints in the documentation of the `Installation `_ +Note that if you place ``requirements.txt`` in the ``docker-context-files`` folder, it will be +used to install all requirements declared there. It is recommended that the file +contains specified version of dependencies to add with ``==`` version specifier, to achieve +stable set of requirements, independent if someone releases a newer version. However you have +to make sure to update those requirements and rebuild the images to account for latest security fixes. + Examples of image customizing ----------------------------- @@ -404,13 +529,13 @@ Such customizations are independent of the way how airflow is installed. The following - rather complex - example shows capabilities of: - * Adding airflow extras (slack, odbc) - * Adding PyPI dependencies (``azure-storage-blob, oauth2client, beautifulsoup4, dateparser, rocketchat_API,typeform``) - * Adding custom environment variables while installing ``apt`` dependencies - both DEV and RUNTIME - (``ACCEPT_EULA=Y'``) - * Adding custom curl command for adding keys and configuring additional apt sources needed to install - ``apt`` dependencies (both DEV and RUNTIME) - * Adding custom ``apt`` dependencies, both DEV (``msodbcsql17 unixodbc-dev g++) and runtime msodbcsql17 unixodbc git procps vim``) +* Adding airflow extras (slack, odbc) +* Adding PyPI dependencies (``azure-storage-blob, oauth2client, beautifulsoup4, dateparser, rocketchat_API,typeform``) +* Adding custom environment variables while installing ``apt`` dependencies - both DEV and RUNTIME + (``ACCEPT_EULA=Y'``) +* Adding custom curl command for adding keys and configuring additional apt sources needed to install + ``apt`` dependencies (both DEV and RUNTIME) +* Adding custom ``apt`` dependencies, both DEV (``msodbcsql17 unixodbc-dev g++) and runtime msodbcsql17 unixodbc git procps vim``) .. exampleinclude:: docker-examples/customizing/custom-sources.sh :language: bash @@ -466,11 +591,11 @@ security vetting and only use the new packages when they were vetted. On a separate (air-gaped) system, all the PyPI packages can be copied to ``docker-context-files`` where you can build the image using the packages downloaded by passing those build args: - * ``INSTALL_FROM_DOCKER_CONTEXT_FILES="true"`` - to use packages present in ``docker-context-files`` - * ``AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"`` - to not pre-cache packages from PyPI when building image - * ``AIRFLOW_CONSTRAINTS_LOCATION=/docker-context-files/YOUR_CONSTRAINT_FILE.txt`` - to downloaded constraint files - * (Optional) ``INSTALL_MYSQL_CLIENT="false"`` if you do not want to install ``MySQL`` - client from the Oracle repositories. In this case also make sure that your +* ``INSTALL_FROM_DOCKER_CONTEXT_FILES="true"`` - to use packages present in ``docker-context-files`` +* ``AIRFLOW_PRE_CACHED_PIP_PACKAGES="false"`` - to not pre-cache packages from PyPI when building image +* ``AIRFLOW_CONSTRAINTS_LOCATION=/docker-context-files/YOUR_CONSTRAINT_FILE.txt`` - to downloaded constraint files +* (Optional) ``INSTALL_MYSQL_CLIENT="false"`` if you do not want to install ``MySQL`` + client from the Oracle repositories. In this case also make sure that your Note, that the solution we have for installing python packages from local packages, only solves the problem of "air-gaped" python installation. The Docker image also downloads ``apt`` dependencies and ``node-modules``. @@ -508,7 +633,7 @@ There are a few things to remember when you modify the ``Dockerfile``: and only the required folders are added through exclusion (!). This allows to keep docker context small because there are many binary artifacts generated in the sources of Airflow and if they are added to the context, the time of building the image would increase significantly. If you want to add any new - folders to be available in the image you must add it here with leading ``!``. + folders to be available in the image you must add it here with leading ``!`` .. code-block:: text diff --git a/docs/docker-stack/docker-examples/customizing/add-build-essential-custom.sh b/docs/docker-stack/docker-examples/customizing/add-build-essential-custom.sh index 230ff1e5cdd2d..05004593a67f3 100755 --- a/docs/docker-stack/docker-examples/customizing/add-build-essential-custom.sh +++ b/docs/docker-stack/docker-examples/customizing/add-build-essential-custom.sh @@ -28,6 +28,6 @@ docker build . \ --build-arg ADDITIONAL_PYTHON_DEPS="mpi4py" \ --build-arg ADDITIONAL_DEV_APT_DEPS="libopenmpi-dev" \ --build-arg ADDITIONAL_RUNTIME_APT_DEPS="openmpi-common" \ - --tag "$(basename "$0")" + --tag "my-build-essential-image:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-build-essential-image:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/custom-sources.sh b/docs/docker-stack/docker-examples/customizing/custom-sources.sh index 22223c4682ca7..8f087b3f3eb2d 100755 --- a/docs/docker-stack/docker-examples/customizing/custom-sources.sh +++ b/docs/docker-stack/docker-examples/customizing/custom-sources.sh @@ -43,6 +43,6 @@ docker build . -f Dockerfile \ curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \ --build-arg ADDITIONAL_RUNTIME_APT_ENV="ACCEPT_EULA=Y" \ --build-arg ADDITIONAL_RUNTIME_APT_DEPS="msodbcsql17 unixodbc git procps vim" \ - --tag "$(basename "$0")" + --tag "my-custom-sources-image:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-custom-sources-image:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/github-different-repository.sh b/docs/docker-stack/docker-examples/customizing/github-different-repository.sh index 35e685e3ad19e..b38ebda91ea06 100755 --- a/docs/docker-stack/docker-examples/customizing/github-different-repository.sh +++ b/docs/docker-stack/docker-examples/customizing/github-different-repository.sh @@ -26,6 +26,6 @@ docker build . \ --build-arg AIRFLOW_INSTALLATION_METHOD="https://github.com/potiuk/airflow/archive/main.tar.gz#egg=apache-airflow" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \ --build-arg CONSTRAINTS_GITHUB_REPOSITORY="potiuk/airflow" \ - --tag "$(basename "$0")" + --tag "github-different-repository-image:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "github-different-repository-image:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/github-master.sh b/docs/docker-stack/docker-examples/customizing/github-main.sh similarity index 94% rename from docs/docker-stack/docker-examples/customizing/github-master.sh rename to docs/docker-stack/docker-examples/customizing/github-main.sh index 3ce40acaf8775..ed1dc368f6575 100755 --- a/docs/docker-stack/docker-examples/customizing/github-master.sh +++ b/docs/docker-stack/docker-examples/customizing/github-main.sh @@ -26,6 +26,6 @@ docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg AIRFLOW_INSTALLATION_METHOD="https://github.com/apache/airflow/archive/main.tar.gz#egg=apache-airflow" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \ - --tag "$(basename "$0")" + --tag "my-github-main:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-github-main:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/github-v2-1-test.sh b/docs/docker-stack/docker-examples/customizing/github-v2-1-test.sh index 6ec055843907e..b8516dd55b5a7 100755 --- a/docs/docker-stack/docker-examples/customizing/github-v2-1-test.sh +++ b/docs/docker-stack/docker-examples/customizing/github-v2-1-test.sh @@ -26,6 +26,6 @@ docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-buster" \ --build-arg AIRFLOW_INSTALLATION_METHOD="https://github.com/apache/airflow/archive/v2-1-test.tar.gz#egg=apache-airflow" \ --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2-0" \ - --tag "$(basename "$0")" + --tag "my-github-v2-1:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-github-v2-1:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/pypi-dev-runtime-deps.sh b/docs/docker-stack/docker-examples/customizing/pypi-dev-runtime-deps.sh index 7dc43bd84fdc9..32bd1fcfac338 100755 --- a/docs/docker-stack/docker-examples/customizing/pypi-dev-runtime-deps.sh +++ b/docs/docker-stack/docker-examples/customizing/pypi-dev-runtime-deps.sh @@ -29,6 +29,6 @@ docker build . \ --build-arg ADDITIONAL_PYTHON_DEPS="pandas" \ --build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \ --build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless" \ - --tag "$(basename "$0")" + --tag "my-pypi-dev-runtime:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-pypi-dev-runtime:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/pypi-extras-and-deps.sh b/docs/docker-stack/docker-examples/customizing/pypi-extras-and-deps.sh index 20deef27c6512..43731216ced90 100755 --- a/docs/docker-stack/docker-examples/customizing/pypi-extras-and-deps.sh +++ b/docs/docker-stack/docker-examples/customizing/pypi-extras-and-deps.sh @@ -27,6 +27,6 @@ docker build . \ --build-arg AIRFLOW_VERSION="2.0.2" \ --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" \ --build-arg ADDITIONAL_PYTHON_DEPS="oauth2client" \ - --tag "$(basename "$0")" + --tag "my-pypi-extras-and-deps:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-pypi-extras-and-deps:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/pypi-selected-version.sh b/docs/docker-stack/docker-examples/customizing/pypi-selected-version.sh index bc72ac1384827..c8e1f395ee6d6 100755 --- a/docs/docker-stack/docker-examples/customizing/pypi-selected-version.sh +++ b/docs/docker-stack/docker-examples/customizing/pypi-selected-version.sh @@ -25,6 +25,6 @@ cd "${AIRFLOW_SOURCES}" docker build . \ --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \ --build-arg AIRFLOW_VERSION="2.0.2" \ - --tag "$(basename "$0")" + --tag "my-pypi-selected-version:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-pypi-selected-version:0.0.1" diff --git a/docs/docker-stack/docker-examples/customizing/stable-airflow.sh b/docs/docker-stack/docker-examples/customizing/stable-airflow.sh index d3471acc181fe..8037785365b4f 100755 --- a/docs/docker-stack/docker-examples/customizing/stable-airflow.sh +++ b/docs/docker-stack/docker-examples/customizing/stable-airflow.sh @@ -23,6 +23,6 @@ cd "${AIRFLOW_SOURCES}" # [START build] docker build . \ - --tag "$(basename "$0")" + --tag "my-stable-airflow:0.0.1" # [END build] -docker rmi --force "$(basename "$0")" +docker rmi --force "my-stable-airflow:0.0.1" diff --git a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile index f0f056bb52ebc..62de197973833 100644 --- a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile @@ -15,7 +15,7 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] -FROM apache/airflow:2.0.2 +FROM apache/airflow USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile index 938dd5707a1ae..b34fdc9ab3cf8 100644 --- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile @@ -15,7 +15,7 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] -FROM apache/airflow:2.0.2 +FROM apache/airflow USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile index c21d5deafc506..cc2559f79d062 100644 --- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile @@ -15,6 +15,6 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] -FROM apache/airflow:2.0.2 +FROM apache/airflow RUN pip install --no-cache-dir lxml # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile index d748e7b233f9a..c849697859fb2 100644 --- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile @@ -15,7 +15,7 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] -FROM apache/airflow:2.0.2 +FROM apache/airflow COPY --chown=airflow:root test_dag.py /opt/airflow/dags diff --git a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile index 752885212eaba..ba07f6816888f 100644 --- a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile +++ b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile @@ -15,7 +15,7 @@ # This is an example Dockerfile. It is not intended for PRODUCTION use # [START Dockerfile] -FROM apache/airflow:2.0.2 +FROM apache/airflow RUN umask 0002; \ mkdir -p ~/writeable-directory # [END Dockerfile] diff --git a/docs/docker-stack/docker-examples/restricted/restricted_environments.sh b/docs/docker-stack/docker-examples/restricted/restricted_environments.sh index 6ff16d18cc8b8..2546dc63db69c 100755 --- a/docs/docker-stack/docker-examples/restricted/restricted_environments.sh +++ b/docs/docker-stack/docker-examples/restricted/restricted_environments.sh @@ -43,5 +43,6 @@ docker build . \ --build-arg INSTALL_MYSQL_CLIENT="false" \ --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" \ --build-arg INSTALL_FROM_DOCKER_CONTEXT_FILES="true" \ - --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-3.7.txt" + --build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-3.7.txt" \ + --tag my-restricted-environment:0.0.1 # [END build] diff --git a/docs/docker-stack/entrypoint.rst b/docs/docker-stack/entrypoint.rst index 48a774d217cfb..c386a67ead28e 100644 --- a/docs/docker-stack/entrypoint.rst +++ b/docs/docker-stack/entrypoint.rst @@ -120,8 +120,82 @@ takes precedence over the :envvar:`AIRFLOW__CORE__SQL_ALCHEMY_CONN` variable. For newer versions, the ``airflow db check`` command is used, which means that a ``select 1 as is_alive;`` query is executed. This also means that you can keep your password in secret backend. +Waits for celery broker connection +---------------------------------- + +In case Postgres or MySQL DB is used, and one of the ``scheduler``, ``celery``, ``worker``, or ``flower`` +commands are used the entrypoint will wait until the celery broker DB connection is available. + +The script detects backend type depending on the URL schema and assigns default port numbers if not specified +in the URL. Then it loops until connection to the host/port specified can be established +It tries :envvar:`CONNECTION_CHECK_MAX_COUNT` times and sleeps :envvar:`CONNECTION_CHECK_SLEEP_TIME` between checks. +To disable check, set ``CONNECTION_CHECK_MAX_COUNT=0``. + +Supported schemes: + +* ``amqp(s)://`` (rabbitmq) - default port 5672 +* ``redis://`` - default port 6379 +* ``postgres://`` - default port 5432 +* ``mysql://`` - default port 3306 + +Waiting for connection involves checking if a matching port is open. +The host information is derived from the variables :envvar:`AIRFLOW__CELERY__BROKER_URL` and +:envvar:`AIRFLOW__CELERY__BROKER_URL_CMD`. If :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable +is passed to the container, it is evaluated as a command to execute and result of this evaluation is used +as :envvar:`AIRFLOW__CELERY__BROKER_URL`. The :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable +takes precedence over the :envvar:`AIRFLOW__CELERY__BROKER_URL` variable. + +.. _entrypoint:commands: + +Executing commands +------------------ + +If first argument equals to "bash" - you are dropped to a bash shell or you can executes bash command +if you specify extra arguments. For example: + +.. code-block:: bash + + docker run -it apache/airflow:2.1.0-python3.6 bash -c "ls -la" + total 16 + drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . + drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 dags + drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 logs + +If first argument is equal to ``python`` - you are dropped in python shell or python commands are executed if +you pass extra parameters. For example: + +.. code-block:: bash + + > docker run -it apache/airflow:2.1.0-python3.6 python -c "print('test')" + test + +If first argument equals to "airflow" - the rest of the arguments is treated as an airflow command +to execute. Example: + +.. code-block:: bash + + docker run -it apache/airflow:2.1.0-python3.6 airflow webserver + +If there are any other arguments - they are simply passed to the "airflow" command + +.. code-block:: bash + + > docker run -it apache/airflow:2.1.0-python3.6 version + 2.1.0 + +Additional quick test options +----------------------------- + +The options below are mostly used for quick testing the image - for example with +quick-start docker-compose or when you want to perform a local test with new packages +added. They are not supposed to be run in the production environment as they add additional +overhead for execution of additional commands. Those options in production should be realized +either as maintenance operations on the database or should be embedded in the custom image used +(when you want to add new packages). + Upgrading Airflow DB --------------------- +.................... If you set :envvar:`_AIRFLOW_DB_UPGRADE` variable to a non-empty value, the entrypoint will run the ``airflow db upgrade`` command right after verifying the connection. You can also use this @@ -131,7 +205,7 @@ intended only for testing purpose, never use SQLite in production as it has seve comes to concurrency. Creating admin user -------------------- +................... The entrypoint can also create webserver user automatically when you enter it. you need to set :envvar:`_AIRFLOW_WWW_USER_CREATE` to a non-empty value in order to do that. This is not intended for @@ -185,66 +259,24 @@ database and creating an ``admin/admin`` Admin user with the following command: The commands above perform initialization of the SQLite database, create admin user with admin password and Admin role. They also forward local port ``8080`` to the webserver port and finally start the webserver. -Waits for celery broker connection ----------------------------------- - -In case Postgres or MySQL DB is used, and one of the ``scheduler``, ``celery``, ``worker``, or ``flower`` -commands are used the entrypoint will wait until the celery broker DB connection is available. - -The script detects backend type depending on the URL schema and assigns default port numbers if not specified -in the URL. Then it loops until connection to the host/port specified can be established -It tries :envvar:`CONNECTION_CHECK_MAX_COUNT` times and sleeps :envvar:`CONNECTION_CHECK_SLEEP_TIME` between checks. -To disable check, set ``CONNECTION_CHECK_MAX_COUNT=0``. - -Supported schemes: - -* ``amqp(s)://`` (rabbitmq) - default port 5672 -* ``redis://`` - default port 6379 -* ``postgres://`` - default port 5432 -* ``mysql://`` - default port 3306 - -Waiting for connection involves checking if a matching port is open. -The host information is derived from the variables :envvar:`AIRFLOW__CELERY__BROKER_URL` and -:envvar:`AIRFLOW__CELERY__BROKER_URL_CMD`. If :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable -is passed to the container, it is evaluated as a command to execute and result of this evaluation is used -as :envvar:`AIRFLOW__CELERY__BROKER_URL`. The :envvar:`AIRFLOW__CELERY__BROKER_URL_CMD` variable -takes precedence over the :envvar:`AIRFLOW__CELERY__BROKER_URL` variable. - -.. _entrypoint:commands: +Installing additional requirements +.................................. -Executing commands ------------------- +Installing additional requirements can be done by specifying ``_PIP_ADDITIONAL_REQUIREMENTS`` variable. +The variable should contain a list of requirements that should be installed additionally when entering +the containers. Note that this option slows down starting of Airflow as every time any container starts +it must install new packages. Therefore this option should only be used for testing. When testing is +finished, you should create your custom image with dependencies baked in. -If first argument equals to "bash" - you are dropped to a bash shell or you can executes bash command -if you specify extra arguments. For example: +Example: .. code-block:: bash - docker run -it apache/airflow:2.1.0-python3.6 bash -c "ls -la" - total 16 - drwxr-xr-x 4 airflow root 4096 Jun 5 18:12 . - drwxr-xr-x 1 root root 4096 Jun 5 18:12 .. - drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 dags - drwxr-xr-x 2 airflow root 4096 Jun 5 18:12 logs - -If first argument is equal to ``python`` - you are dropped in python shell or python commands are executed if -you pass extra parameters. For example: - -.. code-block:: bash - - > docker run -it apache/airflow:2.1.0-python3.6 python -c "print('test')" - test - -If first argument equals to "airflow" - the rest of the arguments is treated as an airflow command -to execute. Example: - -.. code-block:: bash - - docker run -it apache/airflow:2.1.0-python3.6 airflow webserver - -If there are any other arguments - they are simply passed to the "airflow" command - -.. code-block:: bash + docker run -it -p 8080:8080 \ + --env "_PIP_ADDITIONAL_REQUIREMENTS=lxml==4.6.3 charset-normalizer==1.4.1" \ + --env "_AIRFLOW_DB_UPGRADE=true" \ + --env "_AIRFLOW_WWW_USER_CREATE=true" \ + --env "_AIRFLOW_WWW_USER_PASSWORD_CMD=echo admin" \ + apache/airflow:master-python3.8 webserver - > docker run -it apache/airflow:2.1.0-python3.6 version - 2.1.0 +This method is only available starting from Docker image of Airflow 2.1.1 and above. diff --git a/docs/docker-stack/index.rst b/docs/docker-stack/index.rst index 6ee77d03675aa..46b39dc4b6d6f 100644 --- a/docs/docker-stack/index.rst +++ b/docs/docker-stack/index.rst @@ -38,15 +38,32 @@ Docker Image for Apache Airflow For the ease of deployment in production, the community releases a production-ready reference container image. -The docker image provided (as convenience binary package) in the -`apache/airflow DockerHub `_ is a bare image -that has a few external dependencies and extras installed.. -The Apache Airflow image provided as convenience package is optimized for size, so +The Apache Airflow community, releases Docker Images which are ``reference images`` for Apache Airflow. +Every time a new version of Airflow is released, the images are prepared in the +`apache/airflow DockerHub `_ +for all the supported Python versions. + +You can find the following images there (Assuming Airflow version |version|): + +* ``apache/airflow:latest`` - the latest released Airflow image with default Python version (3.6 currently) +* ``apache/airflow:latest-pythonX.Y`` - the latest released Airflow image with specific Python version +* ``apache/airflow:|version|`` - the versioned Airflow image with default Python version (3.6 currently) +* ``apache/airflow:|version|-pythonX.Y`` - the versioned Airflow image with specific Python version + +Those are "reference" images. They contain the most common set of extras, dependencies and providers that are +often used by the users and they are good to "try-things-out" when you want to just take airflow for a spin, + +The Apache Airflow image provided as convenience package is optimized for size, and it provides just a bare minimal set of the extras and dependencies installed and in most cases -you want to either extend or customize the image. You can see all possible extras in -:doc:`extra-packages-ref`. The set of extras used in Airflow Production image are available in the -`Dockerfile `_. +you want to either extend or customize the image. You can see all possible extras in :doc:`extra-packages-ref`. +The set of extras used in Airflow Production image are available in the +`Dockerfile `_. + +However, Airflow has more than 60 community-managed providers (installable via extras) and some of the +default extras/providers installed are not used by everyone, sometimes others extras/providers +are needed, sometimes (very often actually) you need to add your own custom dependencies, +packages or even custom providers. You can learn how to do it in :ref:`Building the image `. The production images are build in DockerHub from released version and release candidates. There are also images published from branches but they are used mainly for development and testing purpose. diff --git a/docs/docker-stack/recipes.rst b/docs/docker-stack/recipes.rst index 66dbbcdf8eeed..f27ed5143ff08 100644 --- a/docs/docker-stack/recipes.rst +++ b/docs/docker-stack/recipes.rst @@ -41,7 +41,7 @@ Then build a new image. docker build . \ --build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \ - -t my-airflow-image + --tag my-airflow-image:0.0.1 Apache Hadoop Stack installation @@ -67,4 +67,4 @@ Then build a new image. docker build . \ --build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \ - -t my-airflow-image + --tag my-airflow-image:0.0.1 diff --git a/docs/helm-chart/production-guide.rst b/docs/helm-chart/production-guide.rst index ec08212b57dbd..bd61808b39d2a 100644 --- a/docs/helm-chart/production-guide.rst +++ b/docs/helm-chart/production-guide.rst @@ -67,8 +67,31 @@ Depending on the size of you Airflow instance, you may want to adjust the follow # The maximum number of server connections to the result backend database from PgBouncer resultBackendPoolSize: 5 -DAG Files ---------- +Extending and customizing Airflow Image +--------------------------------------- + +The Apache Airflow community, releases Docker Images which are ``reference images`` for Apache Airflow. +However, Airflow has more than 60 community managed providers (installable via extras) and some of the +default extras/providers installed are not used by everyone, sometimes others extras/providers +are needed, sometimes (very often actually) you need to add your own custom dependencies, +packages or even custom providers, or add custom tools and binaries that are needed in +your deployment. + +In Kubernetes and Docker terms this means that you need another image with your specific requirements. +This is why you should learn how to build your own ``Docker`` (or more properly ``Container``) image. + +Typical scenarios where you would like to use your custom image: + +* Adding ``apt`` packages +* Adding ``PyPI`` packages +* Adding binary resources necessary for your deployment +* Adding custom tools needed in your deployment + +See `Building the image `_ for more +details on how you can extend and customize the Airflow image. + +Managing DAG Files +------------------ See :doc:`manage-dags-files`. diff --git a/docs/helm-chart/quick-start.rst b/docs/helm-chart/quick-start.rst index b5f0846ec1506..09fa745c2e458 100644 --- a/docs/helm-chart/quick-start.rst +++ b/docs/helm-chart/quick-start.rst @@ -65,8 +65,17 @@ Run ``kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow`` to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working. -Build a Docker image from your DAGs ------------------------------------ +Extending Airflow Image +----------------------- + +The Apache Airflow community, releases Docker Images which are ``reference images`` for Apache Airflow. +However when you try it out you want to add your own DAGS, custom dependencies, +packages or even custom providers. + +The best way to achieve it, is to build your own, custom image. + +Adding DAGs to your image +......................... 1. Create a project @@ -84,7 +93,7 @@ Build a Docker image from your DAGs .. code-block:: bash - docker build -t my-dags:0.0.1 . + docker build --tag my-dags:0.0.1 . 3. Load the image into kind: @@ -101,3 +110,90 @@ Build a Docker image from your DAGs helm upgrade $RELEASE_NAME --namespace $NAMESPACE \ --set images.airflow.repository=my-dags \ --set images.airflow.tag=0.0.1 + +Adding ``apt`` packages to your image +..................................... + +Example below adds ``vim`` apt package. + +1. Create a project + + .. code-block:: bash + + mkdir my-airflow-project && cd my-airflow-project + cat < Dockerfile + FROM apache/airflow + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + vim \ + && apt-get autoremove -yqq --purge \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + USER airflow + EOM + + +2. Then build the image: + + .. code-block:: bash + + docker build --tag my-image:0.0.1 . + + +3. Load the image into kind: + + .. code-block:: bash + + kind load docker-image my-image:0.0.1 + +4. Upgrade Helm deployment: + + .. code-block:: bash + + helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE \ + --set images.airflow.repository=my-image \ + --set images.airflow.tag=0.0.1 + +Adding ``PyPI`` packages to your image +...................................... + +Example below adds ``lxml`` PyPI package. + +1. Create a project + + .. code-block:: bash + + mkdir my-airflow-project && cd my-airflow-project + cat < Dockerfile + FROM apache/airflow + RUN pip install --no-cache-dir lxml + EOM + + +2. Then build the image: + + .. code-block:: bash + + docker build --tag my-image:0.0.1 . + + +3. Load the image into kind: + + .. code-block:: bash + + kind load docker-image my-image:0.0.1 + +4. Upgrade Helm deployment: + + .. code-block:: bash + + helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE \ + --set images.airflow.repository=my-image \ + --set images.airflow.tag=0.0.1 + +Further extending and customizing the image +........................................... + +See `Building the image `_ for more +details on how you can extend and customize the Airflow image. diff --git a/scripts/in_container/prod/entrypoint_prod.sh b/scripts/in_container/prod/entrypoint_prod.sh index 27d1d90f4b52d..123a2f887f0fc 100755 --- a/scripts/in_container/prod/entrypoint_prod.sh +++ b/scripts/in_container/prod/entrypoint_prod.sh @@ -311,6 +311,22 @@ if [[ -n "${_AIRFLOW_WWW_USER_CREATE=}" ]] ; then create_www_user fi +if [[ -n "${_PIP_ADDITIONAL_REQUIREMENTS=}" ]] ; then + >&2 echo + >&2 echo "!!!!! Installing additional requirements: '${_PIP_ADDITIONAL_REQUIREMENTS}' !!!!!!!!!!!!" + >&2 echo + >&2 echo "WARNING: This is a developpment/test feature only. NEVER use it in production!" + >&2 echo " Instead, build a custom image as described in" + >&2 echo + >&2 echo " https://airflow.apache.org/docs/docker-stack/build.html" + >&2 echo + >&2 echo " Adding requirements at container startup is fragile and is done every time" + >&2 echo " the container starts, so it is onlny useful for testing and trying out" + >&2 echo " of adding dependencies." + >&2 echo + pip install --no-cache-dir --user "${_PIP_ADDITIONAL_REQUIREMENTS=}" +fi + # The `bash` and `python` commands should also verify the basic connections # So they are run after the DB check