Document docker poetry best practices

[mxab]

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

There are several discussions on how to build your final application for a docker image

• Docker multi-stage build Dockerfile best practices #1178
• https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker

Would be great if the poetry team could provide a how to poetry with docker documentation

[bufke]

I would add #1227 to this as well.

[fralik]

I recently described usage of Poetry with private repos.

[michaeloliverx]

I have been playing around with multi-stage builds and came up with this example:

# `python-base` sets up all our shared environment variables
FROM python:3.8.1-slim as python-base

    # python
ENV PYTHONUNBUFFERED=1 \
    # prevents python creating .pyc files
    PYTHONDONTWRITEBYTECODE=1 \
    \
    # pip
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    \
    # poetry
    # https://python-poetry.org/docs/configuration/#using-environment-variables
    POETRY_VERSION=1.0.3 \
    # make poetry install to this location
    POETRY_HOME="/opt/poetry" \
    # make poetry create the virtual environment in the project's root
    # it gets named `.venv`
    POETRY_VIRTUALENVS_IN_PROJECT=true \
    # do not ask any interactive question
    POETRY_NO_INTERACTION=1 \
    \
    # paths
    # this is where our requirements + virtual environment will live
    PYSETUP_PATH="/opt/pysetup" \
    VENV_PATH="/opt/pysetup/.venv"


# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"


# `builder-base` stage is used to build deps + create our virtual environment
FROM python-base as builder-base
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        # deps for installing poetry
        curl \
        # deps for building python deps
        build-essential

# install poetry - respects $POETRY_VERSION & $POETRY_HOME
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

# copy project requirement files here to ensure they will be cached.
WORKDIR $PYSETUP_PATH
COPY poetry.lock pyproject.toml ./

# install runtime deps - uses $POETRY_VIRTUALENVS_IN_PROJECT internally
RUN poetry install --no-dev


# `development` image is used during development / testing
FROM python-base as development
ENV FASTAPI_ENV=development
WORKDIR $PYSETUP_PATH

# copy in our built poetry + venv
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

# quicker install as runtime deps are already installed
RUN poetry install

# will become mountpoint of our code
WORKDIR /app

EXPOSE 8000
CMD ["uvicorn", "--reload", "main:app"]


# `production` image used for runtime
FROM python-base as production
ENV FASTAPI_ENV=production
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH
COPY ./app /app/
WORKDIR /app
CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "main:app"]

[fvdnabee]

@bneijt: thank you for replying. Could you elaborate a bit on your setup? Are you running poetry install when building the development container image (i.e in the Dockerfile development stage)? I'm assuming you start the dev container with the bind mount active and then run poetry install (for the first time or at least in a 'fresh' venv; as the bind mount has hidden any venv previous present in the container image) in the running container? So need for a RUN poetry install Dockerfile statement in the development stage in this case?

This setup is quite different from @michaeloliverx's OP, which is a development container image containing both the non-dev and dev dependencies of the poetry project (personally, I also copy the project package to the development image, so you're not forced to use a bind mount). Note I'm not saying which setup is preferable, as both have their merits.

[thehale]

I've created an updated, streamlined version of this Dockerfile here: https://github.com/thehale/docker-python-poetry

• And successfully used it to build a devcontainer here: https://github.com/thehale/multicounter/tree/main/.devcontainer

You can also access pre-built images for a variety of Python/Poetry version combinations on DockerHub: https://hub.docker.com/r/thehale/python-poetry

[AIGeneratedUsername]

@michaeloliverx Why the directory is named "/opt/"?

    # make poetry install to this location
    POETRY_HOME="/opt/poetry" \

What is the purpose of this? What is wrong with a default Poetry location?

[thehale]

@AIGeneratedUsername having a fixed location for the poetry installation makes it easier to copy the poetry installation between stages in a multi-stage build, even if the default poetry location changes.

[usr-ein]

I made this Gist where I try to keep this updated with:

• The new install.python-poetry.org url instead of the raw.githubusercontent.com URL which doesn't work anymore
• Up to date poetry (1.3.2 instead of 1.0.3 at time of writing)
• Up to date Python (3.9 instead of 3.8.1)
• Minor poetry syntax changes (poetry install --with=dev in the dev image)

[mxab]

@michael0liver do you know if there is the requirement for the python:3.8.1-slim in the production step? Since you copy the .venv isn't do we have two python runtimes installed?

I know in the Dockerfile of python:3.8.1-slim happens a bit more then installing python but still isn't there some redundancy when using the venv? (I don't know that much about venvs)

[michaeloliverx]

@mxab Yes you do. Virtual environments are really just an isolated directory where you can install third party packages without affecting the global site-packages. They contain symlinks to the Python executable's installed on your system.

There are the --symlinks & --copies optional arguments documented here which could change the default behavior but I have never messed with these.

I created a python-poetry-docker-example repo with a more concrete example than above. It can still be improved but maybe it could serve as a reference.

[Mithrandir2k18]

Do you think this could work together with cookiecutter? I opened a feature request for development in docker over there. If that all would work together one could replace pyenv/anaconda with docker and use e.g. VSCode Docker Remote development to seamlessly develop inside docker containers without having to worry about installing and managing multiple python version installations on the developers system.

[n8sty]

The pattern I've used when building Docker images with Poetry is to use it's export functionality, build wheels based on the requirements.txt file, and then COPY or make those wheels available to a final, production ready image. Happy to share an example if that would be useful.

[dantp-ai]

Do share it @n8sty!

[bneijt]

I've started using lock packages instead of requirements files. I had issues with markers and optional packages not being dealt with properly.

A full example is available here: https://github.com/bneijt/poetry-lock-docker

[mxab]

@n8sty I read about this way also in the stackoverflow thread.
Last time I tried it I had problems, since I had path dependencies, not sure if this is resolved yet.

Would be cool to see if we had use cases demonstrate how to handle this also (also git and private repos)

[arlyon]

The pattern I've used when building Docker images with Poetry is to use it's export functionality, build wheels based on the requirements.txt file, and then COPY or make those wheels available to a final, production ready image. Happy to share an example if that would be useful.

it should be noted that this doesn't work with dependencies on non-hashable projects (ie. github repos) which is a pain if you, for example, are waiting for patches to land upstream. will update with any workarounds I find

edit: seems to be enough to export without-hashes:

RUN poetry export -f requirements.txt --without-hashes | /venv/bin/pip install -r /dev/stdin

[bneijt]

I've created a script that will create a new package from your pyproject.toml and poetry.lock resulting in a python package that contains all your requirements: https://pypi.org/project/poetry-lock-package/

[dvf]

What a fantastic Dockerfile. Thanks @michael0liver 🍾

[adamwojt]

@michael0liver What a great build. It really helped me with my URL shortener project https://github.com/adamwojt/ur_l
I expanded upon it and used docker-compose to utilize multi-stage. Anyone feel free to use it, it's MIT too.

[tetienne]

@michael0liver Thx for your great example. I use it for my VS code dev container.
I have just one trouble. What do you suggest when we set up some private pypy repositories in pyproject.toml? We have to pass two env variables at build time: POETRY_HTTP_BASIC_XXX_USERNAME and POETRY_HTTP_BASIC_XXX_PASSWORD, and docker suggest to not use ARG for confidential information.

[michaeloliverx]

@michael0liver Thx for your great example. I use it for my VS code dev container.
I have just one trouble. What do you suggest when we set up some private pypy repositories in pyproject.toml? We have to pass two env variables at build time: POETRY_HTTP_BASIC_XXX_USERNAME and POETRY_HTTP_BASIC_XXX_PASSWORD, and docker suggest to not use ARG for confidential information.

You should use docker --build-arg. These variables only exist within the build stage and will not persist in the built image. Expose these variables inside your CI pipeline (for example as secrets inside a GitHub actions workflow).

[mxab]

In case someone uses the buildkit feature of docker you can also use the secret mount

https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information

You also need to put the # syntax block

# syntax=docker/dockerfile:experimental
FROM python:3.6.9-slim-buster as build
....
# then specify a secret id during the install run
RUN --mount=type=secret,id=poetry_auth,dst=/root/.config/pypoetry/auth.toml poetry install -n --no-dev

# mount in the secret file
docker build --secret id=poetry_auth,src=$HOME/.config/pypoetry/auth.toml

[bneijt]

Yes, I use that as well, I also use a PIP_INDEX_URL env secret to pass along the private repository for the initial poetry install.

[mxab]

On other issue we are facing in our case we use sqlalchemy and alembic.
Although alembic says you should install it in the same venv as your project, in my opinion these are two different applications. Therefore my docker application should have internally two seperated venvs, one for alembics migration run and one for the application itself.

Does anyone else do it like this?

[n8sty]

I think that's a bit of a weird setup. Seems like it'd be conceptually simpler if you had two separate containers, one for running migrations and one for running the application itself. I don't there's anything necessarily wrong with this, but unless there's a really important reason (maybe limiting the ability to use additional functional during migrations?) to have multiple virtual environments in a container, I'd think it best to separate them into dedicated containers or combine them into a single environment.

Since this discussion is within the Poetry project, I don't think it supports this particular workflow particularly well either. I can imagine it working with a bunch of different optional dependency groups, and I've seen some issues relating to adding support for more detailed grouping of dependencies, but as of right now, I don't think the current feature set supports this in a particular seamless way.

[yozachar]

Hi, how do I poetry run inside docker?

Here's my (minimal) docker file:

FROM python:3.10.2-slim-bullseye as base

ENV PYTHONFAULTHANDLER=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONHASHSEED=random \
    PYTHONDONTWRITEBYTECODE=1 \
    # pip:
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    # poetry:
    POETRY_VERSION=1.1.13 \
    POETRY_NO_INTERACTION=1 \
    POETRY_CACHE_DIR='/var/cache/pypoetry' \
    PATH="$PATH:/root/.local/bin"

WORKDIR /waka-readme

# install poetry
# RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
RUN pip install pipx
RUN pipx install "poetry==$POETRY_VERSION"
RUN pipx ensurepath

# install dependencies
COPY pyproject.toml poetry.lock ./
RUN poetry install --no-dev --no-root --no-interaction --no-ansi

# copy and run program
COPY main.py ./
CMD [ "poetry", "run", "python", "-m", "main" ]

The CMD [...] fails with poetry not finding the pyproject.toml, which is weird because it was copied earlier. From what I understand WORKDIR provides execution reference to commands like CMD. Maybe I'm wrong, can someone please help me out?

[bneijt]

I can't immediately see the issue, try running the built container with /bin/bash and inspect the state you see there. Also, consider not running powtry inside the container but instead only your application as a package as described by some other posts in this thread. If that does not work, join the discord server and try to find help there, which is probably a better location to get interactive debugging support. Good luck!

[yozachar]

I can't immediately see the issue

You mean this https://github.com/joe733/joe733/runs/5343692505?check_suite_focus=true#step:3:19 ? It's supposed to be executing at /github/workspace/waka-readme but it's not. I wonder why.

powtry inside the container but instead only your application as a package

Executing poetry run python -m main locally (outside container) totally works.

try running the built container with /bin/bash and inspect the state you see there

It executes the program just fine.

[yozachar]

My bad, should've read the docs.. WORKDIR in GitHub Actions is kind of preset. COPY to / (not ./) simply works.

[caniko]

I didn't expect that this problem was this multi-faceted coming in here! Wow.

Anyway, it made me think a little more outside the box. I landed on using archlinux as my base image. I use chaoti-aur to install the latest poetry from git, I can choose the exact Python version I want thanks to pyenv (made it conveniant by adding this as an env var, PYTHON_VERSION):

FROM archlinux AS builder

RUN mkdir /etc/gnupg && \
    echo "honor-http-proxy" > /etc/gnupg/dirmngr.conf && \
    echo "honor-http-proxy" > /etc/pacman.d/gnupg/dirmngr.conf && \
    pacman-key --init && \
    pacman -Syu --noconfirm && \
    pacman -Sy --noconfirm pyenv base-devel openssl zlib xz

ENV PYTHON_VERSION=3.10.3
RUN pyenv install $PYTHON_VERSION

FROM archlinux

RUN mkdir /etc/gnupg && \
    echo "honor-http-proxy" > /etc/gnupg/dirmngr.conf && \
    echo "honor-http-proxy" > /etc/pacman.d/gnupg/dirmngr.conf && \
    pacman-key --init && \
    pacman-key --recv-key 0706B90D37D9B881 FBA220DFC880C036 --keyserver keyserver.ubuntu.com && \
    pacman-key --lsign-key 0706B90D37D9B881 FBA220DFC880C036 && \
    pacman --noconfirm -U 'https://geo-mirror.chaotic.cx/chaotic-aur/chaotic-'{keyring,mirrorlist}'.pkg.tar.zst' && \
    echo "[multilib]" >> /etc/pacman.conf && \
    echo "Include = /etc/pacman.d/mirrorlist" >> /etc/pacman.conf && \
    echo "[chaotic-aur]" >> /etc/pacman.conf && \
    echo "Include = /etc/pacman.d/chaotic-mirrorlist" >> /etc/pacman.conf && \
    pacman -Syu --noconfirm && \
    pacman -Sy --noconfirm python-cleo-git python-poetry-git pyenv

COPY --from=builder /root/.pyenv/versions/$PYTHON_VERSION /root/.pyenv/versions/$PYTHON_VERSION
RUN pyenv rehash && pyenv global $PYTHON_VERSION

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    \
    PIP_NO_CACHE_DIR=1 \
    PIP_DEFAULT_TIMEOUT=100 \
    \
    POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1

You can install your poetry package after copying it to your WORKDIR by poetry install --no-dev. Flush the poetry cache by poetry cache clear --all; no neat way to do this yet #887.

Advantages of archlinux + poetry in containarized production

• Installing poetry through pacman installs the poetry as root. Poetry installer is rootless by default, which is not production-friendly; it is now! Don't forget to switch to a user wihtout write-access from root! Otherwise, you won't benefit from this safety feature.
• Separation of Python used for production code, and poetry code. Poetry will run on the latest stable version of Python, while production code will run on whatever version it was designed for.
• Less maintanance of dockerfile as the heavylifting for poetry compability is done by archlinux.

poetry stable instead of git, alliviate potential dependency issues in production

Just replace pacman -Sy --noconfirm python-cleo-git python-poetry-git pyenv with pacman -Sy --noconfirm python-poetry pyenv.

[caniko]

This is also a gist now

[max-pfeiffer]

I provide and maintain a docker image which provides Poetry on top of the standard Python image. It's available for the latest three Poetry versions and the latest three Python versions: https://github.com/max-pfeiffer/python-poetry

Also I provide two docker images that you can use as builder base for multistage builds. They also contain an example application which demonstrates multistage builds with Poetry:

• https://github.com/max-pfeiffer/uvicorn-poetry
• https://github.com/max-pfeiffer/uvicorn-gunicorn-poetry

Please check out the Docker files for the best practices I applied there. Please leave a comment or send me a pull request if you have additional suggestions.

[dndn10]

Hi, what's the best practice for installing a project with path dependencies (e.g ../packageB/)? The dependency has it's own dependencies. I'm trying to achieve a cached docker layer that installs all remote dependencies - that way only when pyproject.toml or poetry.lock is changed - it would rebuild that stage.

[mxab]

I copy first the basic dependency stuff and then the real files

# Copy deps
COPY package-b/pyproject.toml package-b/poetry.lock package-b/
RUN mkdir -p package-b/package_b && touch package-b/package_b/__init__.py

COPY my-app/pyproject.toml my-app/poetry.lock my-app
RUN mkdir -p my-app/my_app && touch my-app/my_app/__init__.py
RUN --mount=type=secret,id=poetry_auth,dst=/root/.config/pypoetry/auth.toml poetry install -n --only main

# Copy remaining files
COPY package-b/package_b/ package-b/package_b/
COPY my-app/my_app/ my-app/my_app/

[dndn10]

Hi, thanks for the quick answer. I don’t think it’s a good approach since I would have to

1. manage the path dependencies in now in multiple different places ( pyproject.toml AND dockerfile)

2. path dependencies that have their own path dependencies wouldn’t be covered

[ralbertazzi]

I did try to summarize my own findings on Docker and Poetry on this blog post. Feedback and new suggestions are appreciated!

[dndn10]

Thanks for the blog post, it's great!
One thing that is missing in this post is handling local dependencies (aka path dependencies: e.g ../path/to/my/local/package)

[SimonCW]

Hey @ralbertazzi thanks for the post. It helped a bunch, but I have a question remaining:

Is there a way to install only-root in the builder stage and copy the results over to the following layer?

In your blog, you recommend running pip install poetry && poetry install --without dev && pip uninstall poetry in the runtime stage (production image / the 2nd Docker layer). However, this fails because the pyproject.toml is not copied to the runtime stage. Also, the pip install and uninstall are quite annoying.

[ralbertazzi]

Good point. I suggest then installing the project in the build step - by copying the project's code - as a separate RUN instruction. This way the "main" poetry install instruction won't get invalidated every time you change the code. In pseudo code:

COPY pyproject.toml poetry.lock
RUN poetry install --no-root
COPY your code
RUN poetry install

[dominatorzm]

Thanks, sounds great!

[DrChai]

I just went through your blog, it's detailed and helpful. but I don't think ENV VIRTUAL_ENV=/app/.venv is necessary since once you prepend PATH="/app/.venv/bin:$PATH" python should find expected lib path in site.py that is app/.venv(sys.prefix) + lib/pythonX.Y/site-packages

[jmikedupont2]

HI guys, can we please add docs to the repo and not medium?
It's questionable in an open source project context to be promoting paywall content for how to use the project.

[thehale]

@jmikedupont2 I don't get the impression that Python Poetry is itself "promoting paywall content". Rather, individual users have created their own tooling and blog posts to fill a perceived gap in the official documentation.

If you're looking for a repo, here's the image I open-sourced a while ago: https://github.com/thehale/docker-python-poetry

• By stars, it's the second highest rated Docker image under the python-poetry topic, and is a much smaller image than the highest rated image (which packs in lots of other tools which may/may not be useful to individual use cases).

[jmikedupont2]

Okay I I don't want to complain too much but I'm just mentioning that when I view an open source project and I get links to medium that Everyone likes and this content on medium requires me to pay and they're at and they're blocking me through paywall I find that actually quite bad and I want to comment on that because I don't think the project should be promoting links where the documentation could go in to GitHub and be useful to people

[thehale]

Ah, I misread your post. I thought you were criticizing the project itself.

Rather it appears you are encouraging participants to contribute their "medium articles" as docs to the main repository instead of locking their insights behind a paywall.

Full support!

[ralbertazzi]

👋 creator of the medium article and active contributor to Poetry

With my actions I wasn't in any way suggesting to put documentation on medium. Mine is just yet another tech article that shows some opinionated best practices. I wouldn't expect to find the exact same content in the Poetry documentation to be honest.

As for any open-source project, if you feel affected by something then the best thing you can do is open a PR of your own. Again, I speak as a contributor that started opening PRs that were solving the things from which I felt most impacted. Instead, I think none of you ever considered the idea (at least that's what github tells me), so I don't think you are in the best position to judge.

As for most open source: nobody maintains it as a job. Nobody gets paid for maintaining it. The more contributors the better. If you want changes to happen then open a PR.

[BrendanJM]

@ralbertazzi Thanks for putting that article together - I found it very helpful and used it recently when deploying my app (and learned quite a bit about docker too)!

For what it's worth, and I know this is not necessarily a universally shared opinion, I think it is preferable for documentation to have some notion of "this is a great way to do x with our tool", even if that means desteriziling and injecting some opinions into the documentation, as well as discussing how to do things that may technically be not in scope of tool's core functionality.

While it's not possible to offer documentation for all cases, dockerizing a poetry app is a super common thing users may want to do. While the above blog post does a great job diving into best practices, there are quite a few other posts out there that are, shall we say, less best practice driven. With no mention in the official documentation, it can be difficult for poetry users to identify best practices from crufty blog spam. This is compounded even further when we consider geographic web restrictions, linguistic accessibility, and even blog posts with esoteric or obfuscated code as an attack vector.

While poetry technically has nothing to do with docker, I think (caveat: as a non-maintainer) it would be totally appropriate to have a "if you're going to dockerize a poetry app, this is a great way to do that effectively", perhaps even with some level of commentary to help folks understand why and how of what they are doing (good docs are factual; great docs are educational).

An additional benefit to including this type of guidance in the official docs is that when or if something changes, we don't have to hope the blog posts are updated, or a new one comes out and is easily discoverable. The best practice is instead owned by the community and can be discussed openly and modified as needed.

[nanmu42]

After trial and error, I made some changes to michaeloliverx's dockerfile so that:

• You can exec python or poetry in the container and dependencies work all the time.
• No guessing on venv directory name or moving of .venv directory is needed.

# syntax=docker/dockerfile:1
# Keep this syntax directive! It's used to enable Docker BuildKit

################################
# PYTHON-BASE
# Sets up all our shared environment variables
################################
FROM python:3.9.17-slim as python-base

    # Python
ENV PYTHONUNBUFFERED=1 \
    # pip
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    \
    # Poetry
    # https://python-poetry.org/docs/configuration/#using-environment-variables
    POETRY_VERSION=1.6.1 \
    # make poetry install to this location
    POETRY_HOME="/opt/poetry" \
    # do not ask any interactive question
    POETRY_NO_INTERACTION=1 \
    # never create virtual environment automaticly, only use env prepared by us
    POETRY_VIRTUALENVS_CREATE=false \
    \
    # this is where our requirements + virtual environment will live
    VIRTUAL_ENV="/venv" \
    \
    # Node.js major version. Remove if you don't need.
    NODE_MAJOR=18

# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"

# prepare virtual env
RUN python -m venv $VIRTUAL_ENV

# working directory and Python path
WORKDIR /app
ENV PYTHONPATH="/app:$PYTHONPATH"

# pretrained models cache path. Remove if you don't need.
# ref: https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#caching-models
ENV TRANSFORMERS_CACHE="/opt/transformers_cache/"

################################
# BUILDER-BASE
# Used to build deps + create our virtual environment
################################
FROM python-base as builder-base
RUN apt-get update && \
    apt-get install -y \
    apt-transport-https \
    gnupg \
    ca-certificates \
    build-essential \
    git \
    nano \
    curl

# install Node.js. Remove if you don't need.
RUN mkdir -p /etc/apt/keyrings && \
    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
    echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list && \
    apt-get update && apt-get install -y nodejs

# install poetry - respects $POETRY_VERSION & $POETRY_HOME
# The --mount will mount the buildx cache directory to where
# Poetry and Pip store their cache so that they can re-use it
RUN --mount=type=cache,target=/root/.cache \
    curl -sSL https://install.python-poetry.org | python -

# used to init dependencies
WORKDIR /app
COPY poetry.lock pyproject.toml ./
# install runtime deps to $VIRTUAL_ENV
RUN --mount=type=cache,target=/root/.cache \
    poetry install --no-root --only main
COPY scripts scripts/
COPY my_awesome_ai_project/ my_awesome_ai_project/

# populate Huggingface model cache. Remove if you don't need.
RUN poetry run python scripts/bootstrap.py

# build C dependencies. Remove if you don't need.
RUN --mount=type=cache,target=/app/scripts/vendor \
    poetry run python scripts/build-c-denpendencies.py && \
    cp scripts/lib/*.so /usr/lib

################################
# DEVELOPMENT
# Image used during development / testing
################################
FROM builder-base as development

WORKDIR /app

# quicker install as runtime deps are already installed
RUN --mount=type=cache,target=/root/.cache \
    poetry install --no-root --with test,lint

EXPOSE 8080
CMD ["bash"]


################################
# PRODUCTION
# Final image used for runtime
################################
FROM python-base as production

RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
    apt-get install -y --no-install-recommends \
    ca-certificates && \
    apt-get clean

# copy in our built poetry + venv
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $VIRTUAL_ENV $VIRTUAL_ENV
# copy in our C dependencies. Remove if you don't need.
COPY --from=builder-base /app/scripts/lib/*.so /usr/lib
# copy in pre-populated transformer cache. Remove if you don't need.
COPY --from=builder-base $TRANSFORMERS_CACHE $TRANSFORMERS_CACHE

WORKDIR /app
COPY poetry.lock pyproject.toml ./
COPY my_awesome_ai_project/ my_awesome_ai_project/

EXPOSE 8080
CMD ["python", "my_awesome_ai_project/app.py"]

I wrote a blog post Quick Dockerfile for Python Poetry Projects if you want more details.

Feedbacks would be much appreciated!

[nfantone]

Something I'm not quite following: your base stage activates the virtual env, but the Gist you based your Dockerfile on doesn't. How is the app running in production aware of dependencies there?

[nanmu42]

but the Gist you based your Dockerfile on doesn't.

Yes, the gist has the flaw so that Poetry does not work in the production image. You may try it out.

My version solves this problem.

[nfantone]

I see. Any particular reason why you chose to manually append $VIRTUAL_ENV/bin to $PATH instead of source <venv>/bin/activate?

[thehale]

source commands only affect your current terminal session. So, in Docker, they're effectively a no-op since each command runs in its own "terminal session" of sorts.

Updating the PATH is the equivalent operation which can persist to later parts of the image.

[ahmed-moubtahij]

# populate Huggingface model cache. Remove if you don't need.
RUN poetry run python scripts/bootstrap.py

This looks interesting. What does scripts/bootstrap.py look like?

[bhaktatejas922]

I cant seem to get package caching to work. Are multi stage builds needed for this to work?

I'm using github actions with docker

[thehale]

Caching in GitHub Actions is separate from the poetry tool itself.

I typically use the official setup-python which has built-in support for package caching with Poetry.

[supreethmanyam]

There seems to be a workaround for caching mounts in Github Actions.

[BroFromSpace]

Hello everyone. Can anyone help me with my Dockerfile for project with poetry src layout. I tried different approaches, but looks like a stacked with this error "ModuleNotFoundError: No module named 'simple_queueing_system'". I would really appreciate your help. Thanks!

Dockerfile:
https://pastebin.com/428vmwue

[mheguy-flo]

I don't see where you install simple_queueing_system, and I don't see anywhere that indicates it will be in your python path.

[BroFromSpace]

Thanks for your answer, however, I'm not sure what you mean under "install simple_queueing_system" and is pythonpath required, as I didn't see it in examples.

[pgronkievitz]

it's not dockerfile issue, you either haven't copied your simple_queueing_system module into the container or you miss some dependencies in the pyproject.toml and thus you don't have simple_queueing_system in your lock

[mheguy-flo]

I'll phrase things a different way. You are getting a ModuleNotFoundError because Python doesn't know what the name simple_queueing_system means / points to.

uvicorn is taking main:app and translating that into a file named main.py and an attribute in it called app. So it is loading that file perfectly fine. But somewhere in your project, you are importing (or importing from) simple_queueing_system. And that name has no meaning to the import system. It has no meaning because it's not installed in your virtual environment (it's not in the site-packages folder).

[BroFromSpace]

Looks like I figured out how to fix my issue. However, I now got the problem with poetry installing venv. Looks like poetry ignoring the /opt/pysetup folder, however it works fine with /opt/folder-name. Any idea why this happens?

[damonmaria]

With poetry install there is the --compile option. This definitely speeds up startup time. We are using it in our dockerfile but I don't see it mentioned above.

Is there a potential issue with --compile? Like the compilation might differ based on the CPU spec and therefore you'd get different or suboptimal results if build environment is different to the runtime environment of the container?

[mheguy-flo]

Nice find! I forgot that Poetry's new installer no longer compiles bytecode.
I'll be adding that to my dockerfiles. :)

poetry install --compile creates python bytecode from the sources. This is something that also happens when you first run source code.
Python bytecode is portable across operating systems, but not across versions of Python.
Most importantly: bytecode files are marked with the version they're supposed to run with. If it doesn't match what the interpreter is expecting, it will just re-compile the bytecode. :)

[nick-youngblut]

Is this the only official documentation on installing and setting up poetry via a Dockerfile?

It would be great to have >=1 example Dockerfile on best practices for using poetry with a Dockerfile (e.g., a Dockerfile for a VS Code devcontainer). I would like to switch to poetry, but I'm not sure how to best modify my current Dockerfile setup that I use for my devcontainer environments:

# Use an official Python runtime as a base image
FROM python:3.11-slim-buster

# Set the working directory in the container
WORKDIR /my_app

# Install system dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc curl gnupg2 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Copy the requirements.txt file into the container
COPY requirements.txt ./

# Copy the package files into the container
COPY ./my_app ./

# Install any needed Python packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

Maybe something like the following?

# Use an official Python runtime as a base image
FROM python:3.11-slim-buster

# Set the working directory in the container
WORKDIR /my_app

# Install system dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc curl gnupg2 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install Poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

# Add Poetry to PATH
ENV PATH="${PATH}:/root/.local/bin"

# Copy the pyproject.toml (and optionally poetry.lock) file into the container
COPY pyproject.toml poetry.lock* ./

# Configure Poetry: Do not create a virtual environment as the container itself provides isolation
RUN poetry config virtualenvs.create false

# Install dependencies using Poetry
RUN poetry install --no-dev

# Copy the package files into the container
COPY ./my_app ./

[thehale]

Here's the Poetry devcontainer Dockerfile I use in several of my projects: https://github.com/thehale/git-authorship/tree/master/.devcontainer

[fralik]

@nick-youngblut , btw the way you treat requirements file and source code in your dev container makes little sense. There are separate copy commands in order to have pip install in between, not after them. Right now, you will reinstall all python dependencies every time something changes in myapp.

[nick-youngblut]

Thanks @thehale for the link to the dockerfile and devcontainer.json!

In many cases, users will not want (or be able) to use thehale/python-poetry:1.3.2-py3.8-slim as the base image. Do you have an example of using python:3.9-slim-buster or similar?

Are there plans to add an example Dockerfile to the docs?

[thehale]

Do you have an example of using python:3.9-slim-buster or similar?

I currently publish tags on DockerHub for every supported version of Python and Poetry. So far, I've only had requests for the -slim variants, but it's really easy to support others.

To build the base Python/Poetry image yourself, you can use my docker-python-poetry project.

# For `thehale/docker-python-poetry`
make build-version \
    POETRY_VERSION="1.3.2" \
    PYTHON_IMAGE_TAG="3.9-slim-buster"

I'd also happily accept a PR adding slim-buster to the build matrix to include that variant in the auto-published tags.

---

Are there plans to add an example Dockerfile to the docs?

Was this question for me? If so, which docs are you referring to?

[nick-youngblut]

Thanks for pointing out the tags on DockerHub. That's great!

Was this question for me? If so, which docs are you referring to?

I originally created this post because I didn't find anything in the official poetry docs on integrating poetry into a Docker container (e.g., for a VS Code devcontainer). I was just curious whether you were thinking of adding such info.

[ryukyi]

Pasting this here because I couldn't find a small image example:

https://gist.github.com/ryukyi/5836f584039b1ad4fb12fec089de577e

# Example flask app folder structure
#
# ├── app
# │   ├── app.py
# │   └── __init__.py
# ├── Dockerfile
# ├── .dockerignore
# ├── poetry.lock
# └── pyproject.toml

# alpine is a small image
FROM python:3.10-alpine

# Don't run as root
RUN adduser -D appuser && \
    # Set the working directory in the container
    mkdir -p /home/appuser/app && \
    chown appuser:appuser /home/appuser/app && \
    # Use wget instead of curl since curl is external package in alpine
    # https://python-poetry.org/docs/#installation
    wget -O get-poetry.py https://install.python-poetry.org && \
    POETRY_HOME=/home/appuser/.poetry python3 get-poetry.py && \
    rm get-poetry.py

USER appuser

# Add Poetry to PATH
ENV PATH="/home/appuser/.poetry/bin:${PATH}"

# Install dependencies
COPY poetry.lock pyproject.toml ./
RUN poetry install

# Copy the rest of your application into the Docker image
COPY ./app /home/appuser/app

# Run your Flask application using Gunicorn server for production
CMD ["poetry", "run", "gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "app:app"]

[Olegt0rr]

This example does not respect possibility of building libraries written in the C language. That's why top rated answer contains multistage build.

And as far as I remember, Python does not behave very well on Alpine: the recommended thin build is -slim, based on Debian

[clintonroy]

It is almost impossible to give a good recipe that's good for everyone, as how poetry is installed is up to the user, how the virtual environment is set up is up to the user, what sort of project they're using, is, again, up to the user.

With poetry showing up in upstream repos more and more, installing it specially seems less and less the way to go to me.

[nick-youngblut]

It is almost impossible to give a good recipe that's good for everyone

I would argue that is the case for most software configurations, but it would be helpful to have a guide that works for many users. For instance, tutorials do not show the exact workflow that most users need, but they provide a guide and starting point.

With poetry showing up in upstream repos more and more, installing it specially seems less and less the way to go to me.

So how should I install poetry in a Dockerfile that will be used as a developer environment (e.g., a VS Code devcontainer)? You mean to use apt-get, yum, etc.?

[gianfa]

I have read this discussion, which I find very useful, if not necessary.
Thank you for opening it.

I understand that the difficulty now is to converge on an agreement on best practices to write.
I express my opinion below in the hope of doing something useful.

The structure

The following structure could be considered for a dedicated section.

• Best Practices. A brief general introduction on best practices for poetry in Docker, perhaps listed, may help to give a high-level view of the needs to be considered and a reference to the user.
• Use Cases. Then add main use cases, with example Dockerfiles.
• Specifics. Each use case adheres to clearly stated specifications, with specific application.

In general, I would act according to:

• Start small. A use case may also be fine. This way you can collect feedback and over time expand with new use cases.
• Accept that a one-size-fits-all does not exist. If it eventually exists, it will be easy to update the page with such a solution.

Example

Here is an example based on the above ideas

## Introduction

....blabla
The following best practices should be kept in mind

- [optional] set the latest python version, in order to get the latest patch
- [highly suggested] use pip to install poetry
- [critical] never hardcode credentials to private sources
- ...

## Use cases

The following are general use cases that you can use a starting point for your specific case

### UC1: Dev environment

Here is an example of how to create a dev container aimed to host a basic development env. Once the image is built nobody can make OS changes, except the admin. An example of usage is a container used by a team.

#### Specifics

- Unprivileged User. 
- multistage, in order to make the image lighter
- ...

#### Dockerfile

FROM python .......
....