Generated image is missing files generated via RUN

[clemenskol]

Actual behavior

Files generated via a RUN command should be included in the final image (e.g., regardless of file generation timestamp). This seems not to be the case.

I have generated a minimal Dockerfile to demonstrate this:

# syntax=docker/dockerfile:1
FROM amd64/ubuntu as test
RUN \
    apt update && \
    apt install \
        --no-install-recommends \
        --assume-yes \
        python3.11 && \
    ls -l `which python3.11` && \
    python3.11 --version

When building the image, the python3.11 command is not property installed in the generated image, although it's clearly present while building.

My build command:

/kaniko/executor --context /workspace --dockerfile ./Dockerfile --destination <my-repo>:test-tag --snapshot-mode=full --cache=true

The output of the final 2 commands can be seen in the build output:

-rwxr-xr-x 1 root root 6890080 Aug 12  2022 /usr/bin/python3.11
Python 3.11.0rc1

When the generated image is then run, the file is not found (python3.11) simply does not exist.

To test if this has to do with file timestamps, I have done the following modification:

# syntax=docker/dockerfile:1
FROM amd64/ubuntu as test
RUN \
    apt update && \
    apt install \
        --no-install-recommends \
        --assume-yes \
        python3.11 && \
    ls -l `which python3.11` && \
    python3.11 --version && \
    touch `which python3.11`

In this case, the python3.11 binary is in the generated image, but since it's not just the binary itself that is missing (but essentially most files installed via apt), the image is completely non-functional:

docker run --rm -ti <my-repo>:test-tag
root@92196457ce8a:/# python3.11 --version
python3.11: error while loading shared libraries: libexpat.so.1: cannot open shared object file: No such file or directory

Note that I have tried various alternatives using or not using --cache and using different --snapshot-mode

Expected behavior

All files are stored in the generated image.

If I build the image using the Dockerfile above via docker buildx build the image works as expected:

docker run --rm -ti <my-repo>:test-tag
root@93076a150249:/# which python3.11
/usr/bin/python3.11
root@93076a150249:/# python3.11 --version
Python 3.11.0rc1

To Reproduce
Steps to reproduce the behavior:

1. Use Dockerfile above
2. Build using kankiko using the command above
3. Launch image, launch python, see failure (python missing or incorrectly installed)

Additional Information

Kaniko version :  v1.22.0

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[clemenskol]

FYI, I looked at other tickets with a similar problem (e.g., #2336), but either the root-cause described in those tickets is different or the proposed work-around did not work for me.

I have tried many different work-arounds, and none worked for me (aside from touching every file in the file-system, but this is not an option for me)

[clemenskol]

another observation: if I change this to a multi-stage build AND I do more than just use RUN commands, then it sometimes works. Yes, I execute the same build twice in a row, and it seems I have about a 50% chance to get a working container image

Dockerfile:

# syntax=docker/dockerfile:1
FROM amd64/ubuntu as stage
RUN \
    apt update && \
    apt install \
        --no-install-recommends \
        --assume-yes \
        python3.11
FROM stage as final
ADD .ignore /.ignore

Build command (note that I'm not pushing remotely just to safe roundtrip time, pushing to a remote registry has the same outcome):

/kaniko/executor \
    --context /workspace \
    --dockerfile Dockerfile \
    --destination kamiko-test:3 \
    --no-push \
    --tar-path output.tar \
    --target final \
    -v debug

Test command:

docker image rm kamiko-test:3
docker load -i output.tar
docker run --rm -ti kamiko-test:3 bash -c '/usr/bin/python3.11 --version'

In some cases we get this result (installation worked):

Untagged: kamiko-test:3
Deleted: sha256:c42801f9c6b74e0dd7002f9439d0e2675fddc2070665f5646b0303e5e9277a01
Deleted: sha256:58ee2628caa0ebb2dd0b9ee2893bb7f6a3996ed8b41177a209154b270e2952f5
Deleted: sha256:c6a78351595ae2bb76e7284ec47f720e5b7d7e9f66ffab997d24436d143c491d
e2b5084e6f6a: Loading layer [==================================================>]  49.89MB/49.89MB
e74d10928493: Loading layer [==================================================>]     259B/259B
Loaded image: kamiko-test:3
Python 3.11.0rc1

In other cases we get this result (installated files were not committed to the snapshot/image):

Untagged: kamiko-test:3
Deleted: sha256:ac778b382fa91f37cfb3d35e2d56d0a52531fb42082b7e2226e44858b0167f29
Deleted: sha256:a1a681b7fa20e5528304dfe34897ebac67a8f4ff3ecceaf6774445c6fd37fe18
Deleted: sha256:6262b815a55b0dc3bb6679ac18aa94d9aa3fa1074357640627318925a53d05af
e9f9bcb2687e: Loading layer [==================================================>]  6.344MB/6.344MB
f81778963cd0: Loading layer [==================================================>]     252B/252B
Loaded image: kamiko-test:3
bash: line 1: /usr/bin/python3.11: No such file or directory

As said, it's random and about 50% to have the one or other outcome. And even more weirdly, it seems to alternate if it works or if it fails. As if a cache would corrupt and then uncorrupt itself (note that in these experiments the cache is off).

I have captured the stdout (build command outputs) + stderr (kamiko debug-verbosity logs) from a successful and failing build.
The stdout build command output is essentially identical (aside from the download/timing info from apt)
The stderr kamiko debug output is very different however and once contains the expected binary in one of the logs but not the other