Add "--load" to "docker buildx build" commands in Makefile

[rhuss]

What this PR does / why we need it:

For creating Docker images, Docker BuildKit is used. However, "docker buildx build" just store the created image in the build cache so that a subsequent "docker push" won't work.

In order to make targets like make deploy-dev-sklearn make work you either need to specify --push for docker buildx build, or, as in this PR use:

# Build and load the image into the Docker runtime
docker buildx build --load ....
...
# Docker push can only push images that are in the Docker Runtime
docker push

Without this fix I get this error:

cd python && docker buildx build --build-arg BASE_IMAGE=python:3.9-slim-bullseye -t docker.io/rhuss/sklearnserver -f sklearn.Dockerfile .
[+] Building 62.5s (21/21) FINISHED                                                                                        docker-container:xbuilder
 => [internal] load build definition from sklearn.Dockerfile                                                                                    0.0s
 => => transferring dockerfile: 1.53kB                                                                                                          0.0s
....
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
docker push docker.io/rhuss/sklearnserver
Using default tag: latest
The push refers to repository [docker.io/rhuss/sklearnserver]
tag does not exist: rhuss/sklearnserver:latest
make: *** [docker-push-sklearn] Error 1

I wonder how this ever has worked before (except maybe that you have used "docker build" before to load the image into the runtime automatically.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

(no issue yet, but I can open one if required)

Type of changes
Please delete options that are not relevant.

• [ X] Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

make deploy-dev-sklearn works now without an error:

cd python && docker buildx build --load --build-arg BASE_IMAGE=python:3.9-slim-bullseye -t docker.io/rhuss/sklearnserver -f sklearn.Dockerfile .
[+] Building 26.0s (24/24) FINISHED                                                                                        docker-container:xbuilder
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                 0.0s
 => [internal] load build definition from sklearn.Dockerfile                                                                                    0.0s
 => => transferring dockerfile: 1.53kB                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim-bullseye                                                                     1.6s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                   0.0s
 => [internal] load build context                                                                                                               0.0s
 => => transferring context: 20.52kB                                                                                                            0.0s
 => [builder  1/12] FROM docker.io/library/python:3.9-slim-bullseye@sha256:a9dbc6bcdd10b7d174452901747f84cbd711c7a5c3ad16fa4286fbf69c9c6af6     0.0s
 => => resolve docker.io/library/python:3.9-slim-bullseye@sha256:a9dbc6bcdd10b7d174452901747f84cbd711c7a5c3ad16fa4286fbf69c9c6af6               0.0s
 => CACHED [builder  2/12] RUN apt-get update && apt-get install -y --no-install-recommends python3-dev build-essential                         0.0s
 => CACHED [builder  3/12] RUN python3 -m venv /opt/poetry && /opt/poetry/bin/pip install poetry==1.4.0                                         0.0s
 => CACHED [builder  4/12] RUN python3 -m venv /prod_venv                                                                                       0.0s
 => CACHED [builder  5/12] COPY kserve/pyproject.toml kserve/poetry.lock kserve/                                                                0.0s
 => CACHED [builder  6/12] RUN cd kserve && poetry install --no-root --no-interaction --no-cache                                                0.0s
 => CACHED [builder  7/12] COPY kserve kserve                                                                                                   0.0s
 => CACHED [builder  8/12] RUN cd kserve && poetry install --no-interaction --no-cache                                                          0.0s
 => CACHED [builder  9/12] COPY sklearnserver/pyproject.toml sklearnserver/poetry.lock sklearnserver/                                           0.0s
 => CACHED [builder 10/12] RUN cd sklearnserver && poetry install --no-root --no-interaction --no-cache                                         0.0s
 => CACHED [builder 11/12] COPY sklearnserver sklearnserver                                                                                     0.0s
 => CACHED [builder 12/12] RUN cd sklearnserver && poetry install --no-interaction --no-cache                                                   0.0s
 => CACHED [prod 2/6] COPY third_party third_party                                                                                              0.0s
 => CACHED [prod 3/6] RUN useradd kserve -m -u 1000 -d /home/kserve                                                                             0.0s
 => [prod 4/6] COPY --from=builder --chown=kserve:kserve /prod_venv /prod_venv                                                                  3.0s
 => [prod 5/6] COPY --from=builder kserve kserve                                                                                                0.1s
 => [prod 6/6] COPY --from=builder sklearnserver sklearnserver                                                                                  0.0s
 => exporting to docker image format                                                                                                           21.1s
 => => exporting layers                                                                                                                        17.4s
 => => exporting manifest sha256:7748d281a1c7ab329ad0122a75367b4ed24c01942c3210c9455a9586dd5b5a0b                                               0.0s
 => => exporting config sha256:7b3b32fcdc643d05dd7184c2f32274a2f86f7decfa6ffaa6449d9c10b23eb3aa                                                 0.0s
 => => sending tarball                                                                                                                          3.7s
 => importing to docker                                                                                                                         0.0s
docker push docker.io/rhuss/sklearnserver
Using default tag: latest
The push refers to repository [docker.io/rhuss/sklearnserver]
a0ff923410f2: Layer already exists
68027a0c9046: Layer already exists
50bbb9a48135: Layer already exists
a4eca0514297: Layer already exists
29819c6a2c24: Layer already exists
df2aa92a12cd: Layer already exists
c5162a55bbec: Layer already exists
37905cb0bd1e: Layer already exists
87e4e6063607: Layer already exists
039b1ee32061: Layer already exists
latest: digest: sha256:6b78c68ef5762bf9ea7933d0ed131fb1dd5816a498062ccd3a2808f41ac23bec size: 2419
./hack/serving_runtime_image_patch.sh "kserve-sklearnserver.yaml" "docker.io/rhuss/sklearnserver"
clusterservingruntime.serving.kserve.io/kserve-sklearnserver unchanged

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

NONE

[kserve-oss-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rhuss
To complete the pull request process, please assign njhill after the PR has been reviewed.
You can assign the PR to them by writing /assign @njhill in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sivanantha321]

@rhuss What version of docker you are using ? It is working fine for me.

[rhuss]

I'm using Docker Desktop 4.21.1 on macOS 13.5 and an M1 (arm64) chip. For Buildkit, I'm using github.com/docker/buildx v0.11.0 687feca9e8dcd1534ac4c026bc4db5a49de0dd6e

As said, I get a

WARNING: No output specified with docker-container driver. 
Build result will only remain in the build cache. 
To push result image into registry use --push or to load image into docker use --load

which cause the push to fail. Do you see the same warning ?

[rhuss]

@sivanantha321 do you think its better to open an issue first ?

(btw, the push might also "seem" to work if you have already built the image at some point in time with "docker build", but it don't push then the just-built image but an older one).

I think a --load doesn't harm anyways and you are guaranteed to push the proper image that just has been built.

[sivanantha321]

@rhuss No, I am not getting any warnings. I am using Docker Engine 24.0.5 on Ubuntu 22.10 and Buildkit github.com/docker/buildx v0.11.2 9872040. Yes, It would be nice to create an issue. Thanks for the PR :)

cd python && docker buildx build --build-arg BASE_IMAGE=python:3.9-slim-bullseye -t docker.io/sivanantha/sklearnserver -f sklearn.Dockerfile .
[+] Building 253.2s (22/22) FINISHED                                                                                                                                                                                                                 docker:default
 => [internal] load .dockerignore                                                                                                                                                                                                                              0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                0.0s
 => [internal] load build definition from sklearn.Dockerfile                                                                                                                                                                                                   0.1s
 => => transferring dockerfile: 1.53kB                                                                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim-bullseye                                                                                                                                                                                    0.0s
 => [builder  1/12] FROM docker.io/library/python:3.9-slim-bullseye                                                                                                                                                                                            0.0s
 => [internal] load build context                                                                                                                                                                                                                             24.0s
 => => transferring context: 2.99GB                                                                                                                                                                                                                           23.5s
 => [builder  2/12] RUN apt-get update && apt-get install -y --no-install-recommends python3-dev build-essential                                                                                                                                              43.6s
 => [prod 2/6] COPY third_party third_party                                                                                                                                                                                                                    0.8s
 => [prod 3/6] RUN useradd kserve -m -u 1000 -d /home/kserve                                                                                                                                                                                                   0.5s
 => [builder  3/12] RUN python3 -m venv /opt/poetry && /opt/poetry/bin/pip install poetry==1.4.0                                                                                                                                                              28.5s 
 => [builder  4/12] RUN python3 -m venv /prod_venv                                                                                                                                                                                                             3.6s 
 => [builder  5/12] COPY kserve/pyproject.toml kserve/poetry.lock kserve/                                                                                                                                                                                      0.1s 
 => [builder  6/12] RUN cd kserve && poetry install --no-root --no-interaction --no-cache                                                                                                                                                                     48.1s 
 => [builder  7/12] COPY kserve kserve                                                                                                                                                                                                                        20.8s 
 => [builder  8/12] RUN cd kserve && poetry install --no-interaction --no-cache                                                                                                                                                                                1.9s 
 => [builder  9/12] COPY sklearnserver/pyproject.toml sklearnserver/poetry.lock sklearnserver/                                                                                                                                                                 0.1s 
 => [builder 10/12] RUN cd sklearnserver && poetry install --no-root --no-interaction --no-cache                                                                                                                                                              37.2s 
 => [builder 11/12] COPY sklearnserver sklearnserver                                                                                                                                                                                                           0.1s
 => [builder 12/12] RUN cd sklearnserver && poetry install --no-interaction --no-cache                                                                                                                                                                         2.2s
 => [prod 4/6] COPY --from=builder --chown=kserve:kserve /prod_venv /prod_venv                                                                                                                                                                                 3.4s
 => [prod 5/6] COPY --from=builder kserve kserve                                                                                                                                                                                                              27.5s
 => [prod 6/6] COPY --from=builder sklearnserver sklearnserver                                                                                                                                                                                                 0.1s
 => exporting to image                                                                                                                                                                                                                                        18.3s
 => => exporting layers                                                                                                                                                                                                                                       18.2s
 => => writing image sha256:3b19d467a802d7cb52ea9c317bc9e2883ac2ae7c8c16f89f10549f00a266c82d                                                                                                                                                                   0.0s
 => => naming to docker.io/sivanantha/sklearnserver                                                                                                                                                                                                            0.0s
docker push docker.io/sivanantha/sklearnserver
Using default tag: latest
The push refers to repository [docker.io/sivanantha/sklearnserver]
724c1d34711f: Pushed 
2311d419a44f: Pushed 
2311d419a44f: Pushing [==========>                                        ]  141.5MB/646.5MB
3d1501d4fb4a: Pushed 
46126e6197f3: Mounted from sivanantha/image-transformer 
afefa325e27c: Layer already exists 
69d0497b946a: Layer already exists 
2c0f1ba59440: Layer already exists 
8ad0088b4e6d: Layer already exists 
latest: digest: sha256:437b30064517520884adfb5d3ce6ec08f31f426a39e8e9b252d2cabab398e48c size: 2424

[sivanantha321]

@rhuss It seems buildkit is not the default builder for other platforms other than linux. Maybe that's why it is working for me. Set Buildx and BuildKit as the default builder on Linux. moby/moby#43992

[rhuss]

Created #3065 to track the problem.

[rhuss]

Why is the DCO check failing ? I think I have signed this commit properly already.

[sivanantha321]

Why is the DCO check failing ? I think I have signed this commit properly already.

Sign off is missing in your commit message. (e.g) Signed-off-by: Sivanantham Chinnaiyan sivanantham.chinnaiyan@ideas2it.com

[rhuss]

I see, so a GPG signed commit is not good enough, like in

[terrytangyuan]

I cannot reproduce this either.