Help me troubleshoot: why doesn't nerdctl cache npm install?

[zladuric]

Hello,
I have a problem I can't figure out on my own, I wanted to ask for some pointers in troubleshooting my setup.

I have a Node.js repo and application, and I have a Dockerfile set up for it. A minimal repro of the Dockerfile is this:

FROM node:18-bullseye-slim
RUN apt-get update && apt-get upgrade

WORKDIR /opt/app
COPY package.json .
COPY package-lock.json .
RUN npm ci
COPY index.js index.js
CMD ["npm", "start"]

Now, when I build it locally, I get all the steps cached right there on the first run:

[+] Building 0.5s (12/12) FINISHED                                                                                 docker:default
 => [internal] load build definition from Dockerfile                                                                         0.0s
 => => transferring dockerfile: 280B                                                                                         0.0s
 => [internal] load .dockerignore                                                                                            0.0s
 => => transferring context: 2B                                                                                              0.0s
 => [internal] load metadata for docker.io/library/node:18-bullseye-slim                                                     0.4s
 => [1/7] FROM docker.io/library/node:18-bullseye-slim@sha256:f06f9ff9dcc5cfec936a7bf08580c5d9cf0f424fbbd857d3677cff575ff34  0.0s
 => [internal] load build context                                                                                            0.0s
 => => transferring context: 273B                                                                                            0.0s
 => CACHED [2/7] RUN apt-get update && apt-get upgrade                                                                       0.0s
 => CACHED [3/7] WORKDIR /opt/app                                                                                            0.0s
 => CACHED [4/7] COPY package.json .                                                                                         0.0s
 => CACHED [5/7] COPY package-lock.json .                                                                                    0.0s
 => CACHED [6/7] RUN npm ci                                                                                                  0.0s
 => CACHED [7/7] COPY index.js index.js                                                                                      0.0s
 => exporting to image                                                                                                       0.0s
 => => exporting layers                                                                                                      0.0s
 => => writing image sha256:481a2890797586d8bb2437179ef7abf259ccbaca352d83d48a41b1d07a622717                                 0.0s

However, when I build this in my k8s cluster with nerdctl, the RUN npm ci step is not cached.

The other steps are good - in my production dockerfile I have a few more steps (installations of custom libs etc) and they all get cached - but the RUN npm ci is never cached, and always runs anew.

Where could I start to look at the problem here? Why isn't it cached? What's the difference between nerdctl and docker build?

I've searched around the internet with no luck, and my Stackoverflow question isn't getting any responses.

[AkihiroSuda]

when I build this in my k8s cluster with nerdctl

How do you do this?
Is /var/lib/buildkit (or ~/.local/share/buildkit) preserved across builds?

[zladuric]

Oh, it's a somewhat usual thing.
We run k8s on AWS.
We run Jenkins in that k8s cluster (This chart: https://github.com/jenkinsci/helm-charts.git)
We configured Jenkins to spawn pods as additionalAgents.

These additonal Jenkins agents have this specified:

          volumes:
             - type: hostPathVolume 
               hostPath: "/run/containerd/containerd.sock"
               mountPath: "/run/containerd/containerd.sock"
             - type: hostPathVolume
               hostPath: "/run/buildkit/buildkitd.sock"
               mountPath: "/run/buildkit/buildkitd.sock"
          imagePullSecretName: "internal-regcred"

Output you wanted:

/home/jenkins/agent # find / -type d -name "buildkit"
/run/buildkit
/home/jenkins/agent # ls ~/.local/share/buildkit /var/lib/buildkit
ls: /root/.local/share/buildkit: No such file or directory
ls: /var/lib/buildkit: No such file or directory
/home/jenkins/agent # 

Within that Jenkins agent, I run this:

nerdctl build . -f cicd/Dockerfile -t <domain/path>/e2e:latest-5cdc570

Now that I think about it, I don't know if that means that we're sharing this with the Jenkins host, or with the pod host - I think the pod.

But I think it should work - the other commands there are cached, and also, if I shell into the container and run nerdctl build manually, it should cache it between two consequitive runs, shouldn't it?

Does that bit help you help me?

[zladuric]

Strange. I shelled into the container, tried building this manually, the RUN npm ci ran. Then I did it again, and it was cached! But 3 minutes later I logged in to the container again, and it was again NOT cached. And another run right after that, again no cache. Something looks strangely off.

Where could I look into that?

[AkihiroSuda]

Could you report to https://github.com/moby/buildkit/issues ?