SEGV in bud-multiple-platform-no-run test

[edsantiago]

Source: nightly CI run

[+1078s] [not ok 250 bud-multiple-platform-no-run]()
         # (from function `die' in file tests/helpers.bash, line 305,
         #  from function `run_buildah' in file tests/helpers.bash, line 292,
         #  in test file tests/bud.bats, line 3612)
         #   `run_buildah build --signature-policy ${TESTSDIR}/policy.json --jobs=0 --all-platforms --manifest $outputlist -f ${TESTSDIR}/bud/multiarch/Dockerfile.no-run ${TESTSDIR}/bud/multiarch' failed with status 125
         # /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.24.1/tests /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.24.1
         # $ podman-remote build --force-rm=false --layers=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.24.1/tests/policy.json --jobs=0 --all-platforms --manifest localhost/testlist -f /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.24.1/tests/bud/multiarch/Dockerfile.no-run /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.24.1/tests/bud/multiarch
         # time="2022-02-18T10:38:26-06:00" level=error msg="While applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/printcap: no such file or directory"
         # time="2022-02-18T10:38:27-06:00" level=error msg="error unmounting container: error unmounting build container \"\": layer not known"
         # panic: runtime error: invalid memory address or nil pointer dereference
         # [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14a29f2]
         # 
         # goroutine 88 [running]:
         # github.com/containers/buildah/imagebuildah.(*StageExecutor).Execute(0xc0001dcf70, 0x1fc3ae8, 0xc000c39770, 0xc0011fc690, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
         # 	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/buildah/imagebuildah/stage_executor.go:953 +0x2032
         # github.com/containers/buildah/imagebuildah.(*Executor).buildStage(0xc000936800, 0x1fc3ae8, 0xc000c39770, 0xc0011faf90, 0xc0001001e0, 0x2, 0x2, 0x0, 0xc000c1f380, 0x14aac4a, ...)
         # 	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/buildah/imagebuildah/executor.go:484 +0x2cc
         # github.com/containers/buildah/imagebuildah.(*Executor).Build.func3.1(0xc000936800, 0xc000e82610, 0xc000e828be, 0xc0011faf90, 0xc0001001e0, 0x2, 0x2, 0x0, 0xc000dfede0, 0x1fc3ae8, ...)
         # 	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/buildah/imagebuildah/executor.go:679 +0x39a
         # created by github.com/containers/buildah/imagebuildah.(*Executor).Build.func3
         # 	/var/tmp/go/src/github.com/containers/podman/vendor/github.com/containers/buildah/imagebuildah/executor.go:663 +0x26a
         # [linux/s390x] [2/2] STEP 1/2: FROM registry.access.redhat.com/ubi8-micro

See link for full stacktrace, which should make it easy to diagnose.

[cevich]

See also: #3710

[TomSweeneyRedHat]

Probably a dumb question, but I noted that the image we're pulling in the Dockerfile is from DockerHub. Could that be a potential issue? Although I'd think we'd get a "rate Limit" type of error instead.

# A base image that is known to be a manifest list.
FROM docker.io/library/alpine
COPY Dockerfile.no-run /root/
# A different base image that is known to be a manifest list, supporting a
# different but partially-overlapping set of platforms.
FROM registry.access.redhat.com/ubi8-micro
COPY --from=0 /root/Dockerfile.no-run /root/

[flouthoc]

SEGSV can be removed by doing a nil check but i think this is the root cause and being discussed here -> #3710 (comment)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc what is going on with this issue?

[flouthoc]

@rhatdan I lost track of this but picking this again and working on fixing the root here: #3710 (comment)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[cevich]

I haven't seen this in recent memory in any of the nightly jobs. Though @edsantiago maybe has actual data from a bigger dataset?

[edsantiago]

buildah/tests/bud.bats

Lines 5129 to 5132 in 117e97d

	# Note: [This is a bug] jobs=1 is intentionally set here since --jobs=0 sets
	# concurrency to maximum which uncovers all sorts of race condition causing
	# flakes in CI. Please put this back to --jobs=0 when https://github.com/containers/buildah/issues/3710
	# is resolved.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc what do you want to do with this one?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago @flouthoc is this still happening?

[edsantiago]

I reenabled the disabled code in #4537 a few weeks ago. We'll find out if it's truly fixed or not.