Speed up Python docker builds using pre-compiled python #7928

jeffwidman · 2023-08-30T09:47:08Z

Our Python docker builds are slow... a typical CI test run for Python
takes ~21 minutes, of which the first ~10 minutes are spent building the
image.

This results in slower local development, slow CI test suites, and slow deployments.

The main culprit is pyenv install which under the covers downloads the
python source and then compiles it locally. Profiling showed that the
download was quick, so even though pyenv supports aria2c, there's
not much to be gained there. Unfortunately, a quick look at the pyenv
issue tracker showed there's no way to to pass pre-compiled artifacts to
pyenv.

For a long time we've bandied about the idea of switching from pyenv
to downloading pre-compiled Pythons. In fact, the GitHub Actions
publishes versions that we could use here:
https://github.com/actions/python-versions/releases

However, we use pyenv local + pyenv exec throughout our Ruby code
for switching to different Python versions. So we thought that it'd take
a week or more to fully migrate away from pyenv.

Today I had to rebuild the python image multiple times, and got so
annoyed that I decided to poke at it a bit.

It turns out that pyenv is simply a shim layer, and as long as
/usr/local/.pyenv/versions/<x.y.z>/bin exists, it will happily pass
commands to anything in that folder.

So I was able to come up with an intermediate solution that speeds the
builds up drastically without requiring a large code refactor.

Running this locally results in the Python download/install/build
step going from ~500 seconds all the way down to ~75 seconds, a savings
of 7 minutes. Given that a full CI run of the python test suite
previously took ~21 minutes, this cuts it by 1/3.

Related but pulling the pre-compiled Python from a different source:

Use pre-compiled Python from official Docker image #7934

jeffwidman · 2023-08-30T09:56:30Z

Whoa, the benefits are even better on the server!

Before: https://github.com/dependabot/dependabot-core/actions/runs/6015676787/job/16318040127?pr=7905 👉 10m 30s
After: https://github.com/dependabot/dependabot-core/actions/runs/6023271199/job/16339680992?pr=7928 👉 2m 4s

jeffwidman

Annotating the changes.

jeffwidman · 2023-08-30T10:06:41Z

python/Dockerfile

@@ -16,6 +16,8 @@ USER root

 # Install *only* the apt packages required for this builder image to build Python.
 # C-libs needed by users to build their Python packages should be installed down below in the final docker image.
+# TODO: not all these packages may be needed now that we've switched from `pyenv install` which compiled from source to
+# downloading / copying pre-compiled python


There's likely further speed/space improvements possibly by dropping some of these packages, but I'd rather do that in a separate PR after letting this one prove itself in production.

Thinking on this more, pre-compiled won't need these packages, but that there's a decent chance other packages that users end up having to build in native C as part of walking their dep tree would need some of these...

But in that case, we should move them below to the "list of packages we install so users can build their native-C-based deps".

Again, probably not something to touch in this PR, but instead in a follow-on PR once I've validated the initial approach in prod.

jeffwidman · 2023-08-30T10:09:16Z

python/Dockerfile

  && rm -rf /var/lib/apt/lists/*

 COPY --chown=dependabot:dependabot python/helpers /opt/python/helpers
 USER root
+# TODO: Now that switched from `pyenv install` which compiled from source to downloading / copying a pre-compiled python
+# we could entirely drop pyenv if we change our ruby code that calls `pyenv exec` to track which version of python to
+# call and uses the full python paths.


A quick grep of pyenv commands showed this actually should be pretty straightforward, since we already have to keep track of when to tell pyenv to switch to a different python version.

Dropping pyenv completely would simplify the Dockerfile and be one less dep we pull in. But again, rather handle that in a separate PR after letting this one marinate a bit.

jeffwidman · 2023-08-30T10:11:04Z

python/Dockerfile

-  && cd /usr/local/.pyenv \
-  && tar czf 3.8.tar.gz versions/$PY_3_8
+  && cd /opt/hostedtoolcache/Python \
+  && tar czf $PY_3_8.tar.gz $PY_3_8


These are tar'd for space reasons... Since this changes the compile source, I double-checked the size, and each one still averaged ~0.5GB uncompressed. Since majority of our runs use the default python (currently 3.11), it's still worth compressing the non-default versions.

jeffwidman · 2023-08-30T10:12:42Z

python/Dockerfile

-  && cd /usr/local/.pyenv \
-  && tar czf 3.10.tar.gz versions/$PY_3_10
+  && cd /opt/hostedtoolcache/Python \
+  && tar czf $PY_3_10.tar.gz $PY_3_10


I hate the lack of DRY across all these, but AFAIK there's no clean way to loop something like this in a multi-stage docker build.

I didn't duplicate the comments across each one... decided that doing it solely on the default 3.11 install sufficed.

jeffwidman · 2023-08-30T10:14:42Z

python/lib/dependabot/python/language_version_manager.rb

+        # pyenv expects the python installation files in the `versions` folder, but the pre-compiled python3 / pip3
+        # expect to reside in the /opt/hostedtoolcache/Python/x.y.z/x64 dir, so need a symlink to make them play nice.
+        SharedHelpers.run_shell_command(
+          "ln -s /opt/hostedtoolcache/Python/#{python_version}/x64 /usr/local/.pyenv/versions/#{python_version}"


This is the one place where it may be more painful to swap architectures, since I'm not sure how we communicate to the Ruby code that it needs x64 vs arm64? We can worry about that later though...

jeffwidman · 2023-08-30T10:15:33Z

python/Dockerfile

+# The pre-compiled Python expects to be installed to this dir
+RUN mkdir -p /opt/hostedtoolcache/Python/$PY_3_11/x64/ \
+  && cd /opt/hostedtoolcache/Python/$PY_3_11/x64/ \
+  # TODO: Add support for arm64 on Ubuntu whenever actions/python-versions adds support for it. Currently not available.


I expect support for this to be added at some point, eg this upstream PR:

Adding ARM64 support for Ubuntu to resolve encountered issue on actions/setup-python actions/python-versions#230

At that point, it'll be easy here in the Dockerfile to make the x64 vs arm64 value dynamic using TARGETARCH...

I can't build this locally on the M1 chip. Did you consider copying out the built Python from the official Docker images like how we're installing Ruby? That way we get the architectures for free without extra scripting, should simplify this part.

I actually did consider it, but for some reason thought that we were trying to move away from copying from official docker images... but a little digging and I see it's the opposite and the Ruby flip was to move away from apt-get towards docker images.

It's a great idea, let me try it.

Turned out to be pretty straightforward:

Use pre-compiled Python from official Docker image #7934

And it's also a bit cleaner as it allowed copying the files directly into /usr/local/.pyenv/versions, no need for any confusing symlinks.

Our Python docker builds are slow... a typical CI test run for Python takes ~21 minutes, of which the first ~10 minutes are spent building the image. This results in slower local development, slow CI test suites, and slow deployments. The main culprit is `pyenv install` which under the covers downloads the python source and then compiles it locally. Profiling showed that the download was quick, so even though `pyenv` supports `aria2c`, there's not much to be gained there. Unfortunately, a quick look at the `pyenv` issue tracker showed [there's no way to to pass pre-compiled artifacts to `pyenv`](https://github.com/orgs/pyenv/discussions/1872). For a long time we've bandied about the idea of switching from `pyenv` to downloading pre-compiled Pythons. In fact, the GitHub Actions publishes versions that we could use here: https://github.com/actions/python-versions/releases However, we use `pyenv local` + `pyenv exec` throughout our Ruby code for switching to different Python versions. So we thought that it'd take a week or more to fully migrate away from `pyenv`. Today I had to rebuild the python image multiple times, and got so annoyed that I decided to poke at it a bit. It turns out that `pyenv` is simply a shim layer, and as long as `/usr/local/.pyenv/versions/<x.y.z>/bin` exists, it will happily pass commands to anything in that folder. So I was able to come up with an intermediate solution that speeds the builds up drastically without requiring a large code refactor. Running this locally results in the Python download/install/build step going from ~500 seconds all the way down to ~75 seconds, a savings of 7 minutes. Given that a full CI run of the python test suite previously took ~21 minutes, this cuts it by 1/3.

jeffwidman · 2023-08-31T00:08:25Z

Closing in favor of #7934 which is a very similar approach, but pulls the sources from the official docker image... benefits of that approach:

Builds are faster by about 15 seconds for some reason
Cleaner because the linking to the C-libs uses more relative links somehow, so I can fully copy it into /usr/local/.pyenv/versions so avoids confusing symlinks
Supports arm out of the box

jeffwidman requested a review from a team as a code owner August 30, 2023 09:47

github-actions bot added the L: python label Aug 30, 2023

jeffwidman commented Aug 30, 2023

View reviewed changes

jeffwidman force-pushed the download-pre-built-python branch from ceecf3d to d2eb235 Compare August 30, 2023 10:31

jeffwidman force-pushed the download-pre-built-python branch from d2eb235 to d55a0f4 Compare August 31, 2023 00:00

jeffwidman mentioned this pull request Aug 31, 2023

Use pre-compiled Python from official Docker image #7934

Merged

jeffwidman closed this Aug 31, 2023

jeffwidman deleted the download-pre-built-python branch August 31, 2023 00:08

jeffwidman mentioned this pull request Aug 31, 2023

Produce smaller Python builds #7891

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up Python docker builds using pre-compiled python #7928

Speed up Python docker builds using pre-compiled python #7928

jeffwidman commented Aug 30, 2023 •

edited

jeffwidman commented Aug 30, 2023 •

edited

jeffwidman left a comment

jeffwidman Aug 30, 2023

jeffwidman Aug 30, 2023

jeffwidman Aug 30, 2023

jeffwidman Aug 30, 2023

jeffwidman Aug 30, 2023

jeffwidman Aug 30, 2023 •

edited

jeffwidman Aug 30, 2023 •

edited

jakecoffman Aug 30, 2023

jeffwidman Aug 30, 2023

jeffwidman Aug 31, 2023

jeffwidman commented Aug 31, 2023

Speed up Python docker builds using pre-compiled python #7928

Speed up Python docker builds using pre-compiled python #7928

Conversation

jeffwidman commented Aug 30, 2023 • edited

jeffwidman commented Aug 30, 2023 • edited

jeffwidman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffwidman Aug 30, 2023 • edited

Choose a reason for hiding this comment

jeffwidman Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffwidman commented Aug 31, 2023

jeffwidman commented Aug 30, 2023 •

edited

jeffwidman commented Aug 30, 2023 •

edited

jeffwidman Aug 30, 2023 •

edited

jeffwidman Aug 30, 2023 •

edited