Skip to content

Commit

Permalink
Add requirements.txt example to "extending docker" (#26663)
Browse files Browse the repository at this point in the history
When extending the image, there was an example of adding individual
packages via PyPI, but not requirements.txt. This PR fixes it.

(cherry picked from commit 25fdb2c)
  • Loading branch information
potiuk authored and jedcunningham committed Sep 26, 2022
1 parent 237f224 commit b68fb09
Show file tree
Hide file tree
Showing 3 changed files with 54 additions and 3 deletions.
34 changes: 31 additions & 3 deletions docs/docker-stack/build.rst
Expand Up @@ -27,7 +27,9 @@ Quick start scenarios of image extending
----------------------------------------

The most common scenarios where you want to build your own image are adding a new ``apt`` package,
adding a new ``PyPI`` dependency and embedding DAGs into the image.
adding a new ``PyPI`` dependency (either individually or via requirements.txt) and embedding DAGs
into the image.

Example Dockerfiles for those scenarios are below, and you can read further
for more complex cases which might involve either extending or customizing the image. You will find
more information about more complex scenarios below, but if your goal is to quickly extend the Airflow
Expand All @@ -46,8 +48,8 @@ switch to the ``root`` user when running the ``apt`` commands, but do not forget
:end-before: [END Dockerfile]


Adding a new ``PyPI`` package
.............................
Adding new ``PyPI`` packages individually
.........................................

The following example adds ``lxml`` python package from PyPI to the image. When adding packages via
``pip`` you need to use the ``airflow`` user rather than ``root``. Attempts to install ``pip`` packages
Expand All @@ -58,6 +60,19 @@ as ``root`` will fail with an appropriate error message.
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]

Adding packages from requirements.txt
.....................................

The following example adds few python packages from ``requirements.txt`` from PyPI to the image.
Note that similarly when adding individual packages, you need to use the ``airflow`` user rather than
``root``. Attempts to install ``pip`` packages as ``root`` will fail with an appropriate error message.

.. exampleinclude:: docker-examples/extending/add-requirement-packages/Dockerfile
:language: Dockerfile
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]


Embedding DAGs
..............

Expand Down Expand Up @@ -363,6 +378,19 @@ The following example adds ``lxml`` python package from PyPI to the image.
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]

Example of adding packages from requirements.txt
................................................

The following example adds few python packages from ``requirements.txt`` from PyPI to the image.
Note that similarly when adding individual packages, you need to use the ``airflow`` user rather than
``root``. Attempts to install ``pip`` packages as ``root`` will fail with an appropriate error message.

.. exampleinclude:: docker-examples/extending/add-requirement-packages/Dockerfile
:language: Dockerfile
:start-after: [START Dockerfile]
:end-before: [END Dockerfile]


Example when writable directory is needed
.........................................

Expand Down
@@ -0,0 +1,21 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.5.0.dev0
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt
# [END Dockerfile]
@@ -0,0 +1,2 @@
lxml
beautifulsoup4

0 comments on commit b68fb09

Please sign in to comment.