error building image: error building stage: failed to get filesystem from image: mkdir /lib64: file exists and other errors

[SohamChakraborty]

Actual behavior
We are trying to build container images in Jenkins. At this point we are using Docker and evaluating kaniko. During a particular build, we are seeing errors like this:

error building image: error building stage: failed to get filesystem from image: mkdir /lib64: file exists

At some other time, the error is:

error building image: error building stage: failed to get filesystem from image: mkdir /lib: file exists

Another time the error was:

�[37mTRAC�[0m[0001] Creating file /lib/x86_64-linux-gnu/libc-2.28.so 
error building image: error building stage: failed to get filesystem from image: chown /lib/x86_64-linux-gnu/libc-2.28.so: no such file or directory

And in another point the error was:

error building image: error building stage: failed to get filesystem from image: chown /bin/tar: no such file or directory

The errors are different at different times and therefore it is very difficult to troubleshoot/pinpoint what is happening.

The builds on Docker go through just fine.

To test whether the behavior is with Jenkins (and NOT kaniko), we tried to build the image locally (in my workstation) and it succeeded.

So this leads me to believe that there is some sort of "issues" in our CI system (Jenkins) that is causing the builds to fail. But the question is why different errors at different times. And to top it all, at one point, the build even went through just fine. But it's somewhat unreliable/erratic behavior.

Apparently this file exists error was fixed as part of this #1469 but somehow (not sure if this is same) the error is still there.

Expected behavior
Either the build will succeed OR the build will fail consistently with same error

To Reproduce
Steps to reproduce the behavior:

1. Not sure whether this can be very easily reproduced, we are running kOps on AWS spot instances. Jenkins version is 2.440.1.
2. Kubernetes plugin is 4199.va_1647c280eb_2.
3. Kaniko image is gcr.io/kaniko-project/executor:v1.22.0-debug.
4. Relevant kaniko-executor command is:

/kaniko/executor --verbosity trace --context `pwd`  \
--dockerfile `pwd`/xx/yy/Dockerfile \
  --destination abc/xyz:${GIT_COMMIT}

Additional Information

• Dockerfile

FROM abc/openjdk:14

ENV APP_ROOT /opt/flyway
ENV FLYWAY_VERSION 9.22.3
ENV FLYWAY_URL https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/${FLYWAY_VERSION}/flyway-commandline-${FLYWAY_VERSION}.tar.gz
ENV ADD_DEFAULT_APPS false
ENV SQL_DIR ${APP_ROOT}/sql
ENV SQLS_LOCATIONS filesystem:${SQL_DIR}
ENV MIGRATIONS_CLASSPATH_LOCATIONS classpath:somedir/common

RUN \
  curl -fSL "$FLYWAY_URL" -o /tmp/flyway.tar.gz && \
  tar -xvf /tmp/flyway.tar.gz -C /opt && \
  mv "${APP_ROOT}-${FLYWAY_VERSION}" "$APP_ROOT" && \
  rm /tmp/flyway.tar.gz

COPY xx/aa ${SQL_DIR}/aa
COPY xx/bb ${SQL_DIR}/bb
...
COPY xx/yy/bin/start.sh /start.sh
COPY xx/yy/jars/*.jar ${APP_ROOT}/jars
RUN chmod a+x /start.sh

CMD ["/start.sh", "migrate"]

• Build Context
  It fails even before COPY commands can be run
• Kaniko Image (fully qualified with digest)
  image: gcr.io/kaniko-project/executor:v1.22.0-debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile