Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding extra requirements for build and runtime of the PROD image. #16170

Merged
merged 1 commit into from Jun 1, 2021
Merged

Adding extra requirements for build and runtime of the PROD image. #16170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 7 additions & 0 deletions Dockerfile
View file
Open in desktop
Expand Up @@ -264,6 +264,13 @@ RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \
find /root/.local -executable -print0 | xargs --null chmod g+x; \
find /root/.local -print0 | xargs --null chmod g+rw

# In case there is a requirements.txt file in "docker-context-files" it will be installed
# during the build additionally to whatever has been installed so far. It is recommended that
# the requirements.txt contains only dependencies with == version specification
RUN if [[ -f /docker-context-files/requirements.txt ]]; then \
pip install --no-cache-dir --user -r /docker-context-files/requirements.txt; \
fi

ARG BUILD_ID
ARG COMMIT_SHA
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
Expand Down
17 changes: 9 additions & 8 deletions IMAGES.rst
View file
Open in desktop
Expand Up @@ -495,7 +495,7 @@ additional apt dev and runtime dependencies.
--build-arg ADDITIONAL_PYTHON_DEPS="pandas"
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++"
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless"
--tag my-image
--tag my-image:0.0.1


the same image can be built using ``breeze`` (it supports auto-completion of the options):
Expand Down Expand Up @@ -533,7 +533,7 @@ based on example in `this comment <https://github.com/apache/airflow/issues/8605
--build-arg ADDITIONAL_RUNTIME_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add --no-tty - && curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="msodbcsql17 unixodbc git procps vim" \
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
--tag my-image
--tag my-image:0.0.1

CI image build arguments
------------------------
Expand Down Expand Up @@ -664,44 +664,45 @@ This builds the CI image in version 3.7 with default extras ("all").

.. code-block:: bash

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster"
docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --tag my-image:0.0.1


This builds the CI image in version 3.6 with "gcp" extra only.

.. code-block:: bash

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_EXTRAS=gcp
--build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1


This builds the CI image in version 3.6 with "apache-beam" extra added.

.. code-block:: bash

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam"
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1

This builds the CI image in version 3.6 with "mssql" additional package added.

.. code-block:: bash

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_PYTHON_DEPS="mssql"
--build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1

This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev dependencies added.

.. code-block::

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++"
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1

This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added.

.. code-block::

docker build . -f Dockerfile.ci --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless"
--build-arg AIRFLOW_EXTRAS=jdbc --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless" \
--tag my-image:0.0.1

CI Image manifests
------------------
Expand Down
4 changes: 2 additions & 2 deletions dev/README_RELEASE_PROVIDER_PACKAGES.md
View file
Open in desktop
Expand Up @@ -601,14 +601,14 @@ USER ${AIRFLOW_UID}
To build an image build and run a shell, run:

```shell script
docker build . -t my-airflow
docker build . --tag my-image:0.0.1
docker run -ti \
--rm \
-v "$PWD/data:/opt/airflow/" \
-v "$PWD/keys/:/keys/" \
-p 8080:8080 \
-e AIRFLOW__CORE__LOAD_EXAMPLES=True \
my-airflow bash
my-image:0.0.1 bash
```

### Additional Verification
Expand Down
4 changes: 2 additions & 2 deletions dev/check_files.py
View file
Open in desktop
Expand Up @@ -47,7 +47,7 @@


DOCKER_CMD = """
docker build -t local/airflow .
docker build --tag local/airflow .
docker local/airflow info
"""

Expand Down Expand Up @@ -80,7 +80,7 @@ def create_docker(txt: str):
print("\n[bold]To check installation run:[/bold]")
print(
"""\
docker build -f Dockerfile.pmc -t local/airflow .
docker build -f Dockerfile.pmc --tag local/airflow .
docker run local/airflow info
"""
)
Expand Down
1 change: 0 additions & 1 deletion docs/apache-airflow/index.rst
View file
Open in desktop
Expand Up @@ -101,7 +101,6 @@ unit of work and continuity.
changelog
best-practices
production-deployment
backport-providers
faq
privacy_notice

Expand Down
26 changes: 16 additions & 10 deletions docs/apache-airflow/start/docker-compose.yaml
View file
Open in desktop
Expand Up @@ -23,16 +23,21 @@
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:|version|
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
# Default: airflow
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:|version|
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
Expand All @@ -50,6 +55,7 @@ x-airflow-common:
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
Expand Down
38 changes: 27 additions & 11 deletions docs/apache-airflow/start/docker.rst
View file
Open in desktop
Expand Up @@ -15,6 +15,8 @@
specific language governing permissions and limitations
under the License.

.. _running-airflow-in-docker:

Running Airflow in Docker
#########################

Expand Down Expand Up @@ -73,7 +75,8 @@ Some directories in the container are mounted, which means that their contents a
- ``./logs`` - contains logs from task execution and scheduler.
- ``./plugins`` - you can put your :doc:`custom plugins </plugins>` here.

This file uses the latest Airflow image (`apache/airflow <https://hub.docker.com/r/apache/airflow>`__). If you need install a new Python library or system library, you can :doc:`customize and extend it <docker-stack:index>`.
This file uses the latest Airflow image (`apache/airflow <https://hub.docker.com/r/apache/airflow>`__).
If you need install a new Python library or system library, you can :doc:`build your image <docker-stack:index>`.

.. _initializing_docker_compose_environment:

Expand Down Expand Up @@ -260,13 +263,26 @@ runtime user id which is unknown at the time of building the image.
| | you want to use different UID than default it must | |
| | be set to ``0``. | |
+--------------------------------+-----------------------------------------------------+--------------------------+
| ``_AIRFLOW_WWW_USER_USERNAME`` | Username for the administrator UI account. | |
| | If this value is specified, admin UI user gets | |
| | created automatically. This is only useful when | |
| | you want to run Airflow for a test-drive and | |
| | want to start a container with embedded development | |
| | database. | |
+--------------------------------+-----------------------------------------------------+--------------------------+
| ``_AIRFLOW_WWW_USER_PASSWORD`` | Password for the administrator UI account. | |
| | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set. | |
+--------------------------------+-----------------------------------------------------+--------------------------+

Those additional variables are useful in case you are trying out/testing Airflow installation via docker compose.
They are not intended to be used in production, but they make the environment faster to bootstrap for first time
users with the most common customizations.

+----------------------------------+-----------------------------------------------------+--------------------------+
| Variable | Description | Default |
+==================================+=====================================================+==========================+
| ``_AIRFLOW_WWW_USER_USERNAME`` | Username for the administrator UI account. | airflow |
| | If this value is specified, admin UI user gets | |
| | created automatically. This is only useful when | |
| | you want to run Airflow for a test-drive and | |
| | want to start a container with embedded development | |
| | database. | |
+----------------------------------+-----------------------------------------------------+--------------------------+
| ``_AIRFLOW_WWW_USER_PASSWORD`` | Password for the administrator UI account. | airflow |
| | Only used when ``_AIRFLOW_WWW_USER_USERNAME`` set. | |
+----------------------------------+-----------------------------------------------------+--------------------------+
| ``_PIP_ADDITIONAL_REQUIREMENTS`` | If not empty, airflow containers will attempt to | |
| | install requirements specified in the variable. | |
| | example: ``lxml==4.6.3 charset-normalizer==1.4.1``. | |
| | Available in Airflow image 2.1.1 and above. | |
+----------------------------------+-----------------------------------------------------+--------------------------+
5 changes: 5 additions & 0 deletions docs/conf.py
View file
Open in desktop
Expand Up @@ -258,10 +258,15 @@ def _get_rst_filepath_from_path(filepath: str):
html_extra_with_substitutions = [
f"{ROOT_DIR}/docs/apache-airflow/start/docker-compose.yaml",
]
# Replace "|version|" in links
manual_substitutions_in_generated_html = [
"installation.html",
]

if PACKAGE_NAME == 'docker-stack':
# Replace "|version|" inside ```` quotes
manual_substitutions_in_generated_html = ["build.html"]

# -- Theme configuration -------------------------------------------------------
# Custom sidebar templates, maps document names to template names.
html_sidebars = {
Expand Down