intermittent wrong architecture selected when building cross platform

[cvaliente]

Actual behavior
kaniko occasionally fails building arm64 images, but succeeds after several retries.
sometimes the build fails in this stage:

INFO[0016] RUN mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/ 
INFO[0016] Initializing snapshotter ...                 
INFO[0016] Taking snapshot of full filesystem...        
INFO[0025] Cmd: /bin/sh                                 
INFO[0025] Args: [-c mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/] 
INFO[0025] Running: [/bin/sh -c mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

if we retry the job, it usually succeeds. Only one out of 5 times actually fails, in an otherwise identical environment.

Expected behavior
A clear and concise description of what you expected to happen.

To Reproduce
We are building two versions, one for arm64, one for amd64. the base image is provided for both architectures, we are running a simple RUN mkdir &&cp in the docker file.

build command:

      /kaniko/executor \
          --dockerfile ${DOCKERFILE} \
          --context ${CI_PROJECT_DIR} \
          --custom-platform ${PLATFORM} \
          --build-arg opts='GOARCH=${GOARCH}' \
          --destination ${DOCKER_REGISTRY}:${VERSION}${SUFFIX}

with

    SUFFIX: "-aarch64"
    PLATFORM: "linux/arm64/v8"
    GOARCH: arm64

Additional Information

• Dockerfile

FROM flink:1.17.2-scala_2.12-java8
RUN mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/

• Build Context
  running on a gitlab cicd runner
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:v1.21.1-debug
• full logs:

$ /kaniko/executor \ # collapsed multi-line command
INFO[0000] Retrieving image manifest flink:1.17.2-scala_2.12-java8 
INFO[0000] Retrieving image flink:1.17.2-scala_2.12-java8 from registry index.docker.io 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest flink:1.17.2-scala_2.12-java8 
INFO[0002] Returning cached image manifest              
INFO[0002] Executing 0 build triggers                   
INFO[0002] Building stage 'flink:1.17.2-scala_2.12-java8' [idx: '0', base-idx: '-1'] 
INFO[0002] Unpacking rootfs as cmd RUN mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/ requires it. 
INFO[0016] RUN mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/ 
INFO[0016] Initializing snapshotter ...                 
INFO[0016] Taking snapshot of full filesystem...        
INFO[0025] Cmd: /bin/sh                                 
INFO[0025] Args: [-c mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/] 
INFO[0025] Running: [/bin/sh -c mkdir ./plugins/s3-fs-hadoop && cp ./opt/flink-s3-fs-hadoop-*.jar ./plugins/s3-fs-hadoop/] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[ricardorqr]

Any update on this problem? I'm having the same issues.