Cannot run `podman farm build` with podman-remote #22647

tnk4on · 2024-05-08T20:40:12Z

Issue Description

I want to run podman farm on a macOS podman machine(arm64) in combination with an amd64 node.

podman farm build command succeeds up to the building of the image, but it fails in the push phase of manifest, regardless of whether the repository to push to exists or not.

Steps to reproduce the issue

My environment

% podman system connection ls
Name                         URI                                                          Identity                                                        Default     ReadWrite
podman-machine-default       ssh://core@127.0.0.1:61793/run/user/501/podman/podman.sock   /Users/shtanaka/.local/share/containers/podman/machine/machine  true        true
podman-machine-default-root  ssh://root@127.0.0.1:61793/run/podman/podman.sock            /Users/shtanaka/.local/share/containers/podman/machine/machine  false       true
rhel94-amd                   ssh://user@192.168.1.32:22/run/user/1000/podman/podman.sock  /Users/shtanaka/.ssh/id_ed25519                                 false       true
% podman farm ls
Name        Connections   Default     ReadWrite
fm1         [rhel94-amd]  true        true

If the repository exists

% cat Containerfile
FROM alpine
RUN arch | tee arch.txt

% podman farm build -t quay.io/tnk4on/farm-test --jobs 2 .
Setting up local builder
Local builder ready
Connecting to "rhel94-amd"
Builder "rhel94-amd" ready
Farm "fm1" ready
Starting build for [{linux arm64 v8}] at "(local)"
Starting build for [{linux amd64 }] at "fm1"
[linux/arm64/v8@(local)] STEP 1/2: FROM alpine
[linux/amd64@fm1] STEP 1/2: FROM alpine
[linux/arm64/v8@(local)] STEP 2/2: RUN arch | tee arch.txt
[linux/arm64/v8@(local)] --> Using cache 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
[linux/arm64/v8@(local)] --> 841a41e43658
[linux/arm64/v8@(local)] 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
finished build for [{linux arm64 v8}] at "(local)": built 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
[linux/amd64@fm1] STEP 2/2: RUN arch | tee arch.txt
[linux/amd64@fm1] --> Using cache 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
[linux/amd64@fm1] --> 729ad4af7256
[linux/amd64@fm1] 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
finished build for [{linux amd64 }] at "fm1": built 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
Getting image source signatures
Copying blob sha256:4d8ae53457511d9767d21fd524bfcce4656a832df587184d7d6ae23bc9138751
Copying blob sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
Getting image source signatures
Copying blob sha256:c04f0bd15842951596d635a4aec2298cefd32a6718927a793467013c731c7a1a
Copying blob sha256:b09314aec293bcd9a8ee5e643539437b3846f9e5e55f79e282e5f67e3026de5e
Copying config sha256:729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
Writing manifest to image destination
Copying config sha256:841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
Writing manifest to image destination
Error: build: error clearing list "quay.io/tnk4on/farm-test"

If the repository does not exist

% podman farm build -t quay.io/tnk4on/farm-test --jobs 2 .
Setting up local builder
Local builder ready
Connecting to "rhel94-amd"
Builder "rhel94-amd" ready
Farm "fm1" ready
Starting build for [{linux arm64 v8}] at "(local)"
Starting build for [{linux amd64 }] at "fm1"
[linux/amd64@fm1] STEP 1/2: FROM alpine
[linux/arm64/v8@(local)] STEP 1/2: FROM alpine
[linux/amd64@fm1] STEP 2/2: RUN arch | tee arch.txt
[linux/amd64@fm1] --> Using cache 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
[linux/amd64@fm1] --> 729ad4af7256
[linux/amd64@fm1] 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
finished build for [{linux amd64 }] at "fm1": built 729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e
[linux/arm64/v8@(local)] STEP 2/2: RUN arch | tee arch.txt
[linux/arm64/v8@(local)] --> Using cache 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
[linux/arm64/v8@(local)] --> 841a41e43658
[linux/arm64/v8@(local)] 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
finished build for [{linux arm64 v8}] at "(local)": built 841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
Getting image source signatures
Copying blob sha256:4d8ae53457511d9767d21fd524bfcce4656a832df587184d7d6ae23bc9138751
Copying blob sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
Getting image source signatures
Copying blob sha256:c04f0bd15842951596d635a4aec2298cefd32a6718927a793467013c731c7a1a
Copying blob sha256:b09314aec293bcd9a8ee5e643539437b3846f9e5e55f79e282e5f67e3026de5e
Copying config sha256:841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271
Writing manifest to image destination
Error: build: building: 2 errors occurred:
	* pushing image {"729ad4af7256bd5afcfc3b309ce3a6e844dc388de0e7312814919b6650e2a68e" "oci-archive"} to registry: writing blob: initiating layer upload to /v2/tnk4on/farm-test/blobs/uploads/ in quay.io: unauthorized: access to the requested resource is not authorized
	* pushing image {"841a41e43658ac9b1fb557edec35e5d9151b45f042bf164e9f29b583d8115271" "oci-archive"} to registry: writing manifest: uploading manifest sha256:25b0796d8bb8ad24c8703345c0e38850bc8e2e6d86bf28a843d75546fe2c543e to quay.io/tnk4on/farm-test: manifest invalid

Describe the results you received

Writing manifest to image destination
Error: build: error clearing list "quay.io/tnk4on/farm-test"

Describe the results you expected

Writing manifest list to image destination
Storing list signatures
Saved list to "quay.io/tnk4on/farm-test"

podman info output

% podman info
host:
  arch: arm64
  buildahVersion: 1.36.0-dev
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.78
    systemPercent: 0.06
    userPercent: 0.16
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.7.9-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 576704512
  memTotal: 2047799296
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240329131657512331.main.31.g2c315a1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240329131649297909.main.62.gad066d4.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240402154251979914.main.23.ga32cc45.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 8d930647dc4551bddb633fa1c8137dc3ce9b08e0
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.aarch64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 18h 34m 57.00s (Approximately 0.75 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 5382139904
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.0-dev-ff7a3dc35
  Built: 1712102400
  BuiltTime: Wed Apr  3 09:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.0-dev-ff7a3dc35

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

% podman version
Client:       Podman Engine
Version:      5.1.0-dev
API Version:  5.1.0-dev
Go Version:   go1.22.2
Git Commit:   3718c197c90f497b805039cd8efad55417ce5fdc
Built:        Thu Apr 18 10:39:38 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.1.0-dev-ff7a3dc35
API Version:  5.1.0-dev-ff7a3dc35
Go Version:   go1.21.8
Built:        Wed Apr  3 09:00:00 2024
OS/Arch:      linux/arm64
% sw_vers
ProductName:		macOS
ProductVersion:		14.4.1
BuildVersion:		23E224

Additional information

Alternatives
Running the same command from inside the podman machine succeeds.

% podman machine ssh
core@localhost:~$ podman system connection ls
Name        URI                                                          Identity                         Default     ReadWrite
rhel94-amd  ssh://user@192.168.1.32:22/run/user/1000/podman/podman.sock  /Users/shtanaka/.ssh/id_ed25519  false       true
core@localhost:~$ podman farm ls
Name        Connections   Default     ReadWrite
fm1         [rhel94-amd]  true        true
core@localhost:~$ cat Containerfile
FROM alpine
RUN arch > arch.txt
core@localhost:~$ podman farm build -t quay.io/tnk4on/farm-test --jobs 2 .
Setting up local builder
Local builder ready
Connecting to "rhel94-amd"
Builder "rhel94-amd" ready
Farm "fm1" ready
Starting build for [{linux amd64 }] at "fm1"
Starting build for [{linux arm64 v8}] at "(local)"
[linux/arm64/v8@(local)] STEP 1/2: FROM alpine
[linux/arm64/v8@(local)] STEP 2/2: RUN arch > arch.txt
[linux/arm64/v8@(local)] --> Using cache 2fbccc56046c73d31d6cf82bd4b83172540f8c9aaddcf5cc4a8876de4e0076ab
[linux/arm64/v8@(local)] --> 2fbccc56046c
[linux/arm64/v8@(local)] 2fbccc56046c73d31d6cf82bd4b83172540f8c9aaddcf5cc4a8876de4e0076ab
finished build for [{linux arm64 v8}] at "(local)": built 2fbccc56046c73d31d6cf82bd4b83172540f8c9aaddcf5cc4a8876de4e0076ab
[linux/amd64@fm1] STEP 1/2: FROM alpine
[linux/amd64@fm1] STEP 2/2: RUN arch > arch.txt
[linux/amd64@fm1] --> Using cache fde5c3cd38cc5c52d29e99669f31d615dc0dcb4aca0232c330181be269c1009a
[linux/amd64@fm1] --> fde5c3cd38cc
[linux/amd64@fm1] fde5c3cd38cc5c52d29e99669f31d615dc0dcb4aca0232c330181be269c1009a
finished build for [{linux amd64 }] at "fm1": built fde5c3cd38cc5c52d29e99669f31d615dc0dcb4aca0232c330181be269c1009a
Getting image source signatures
Getting image source signatures
Copying blob sha256:d9d99f5e5b66a261c507a7ef83122ad473c295ce6e071e390ae1493c2d5cdc08
Copying blob sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
Copying blob 280bbb631e50 done   |
Copying blob 280bbb631e50 done   |
Copying blob 280bbb631e50 done   |
Copying blob b09314aec293 done   |
Copying config 2fbccc5604 done   |
Writing manifest to image destination
Getting image list signatures
Copying 0 images generated from 2 images in list
Writing manifest list to image destination
Storing list signatures
Saved list to "quay.io/tnk4on/farm-test"

The text was updated successfully, but these errors were encountered:

tnk4on · 2024-05-09T08:51:03Z

I found the cause.
ctx -> ir.ClientCtx

I am working on it and will open a PR.

podman/pkg/domain/infra/tunnel/manifest.go

Lines 202 to 216 in 092d040

    
           // ManifestListClear clears out all instances from a manifest list 
        
           func (ir *ImageEngine) ManifestListClear(ctx context.Context, name string) (string, error) { 
        
           	listContents, err := manifests.InspectListData(ctx, name, &manifests.InspectOptions{}) 
        
           	if err != nil { 
        
           		return "", err 
        
           	} 
        
           	for _, instance := range listContents.Manifests { 
        
           		if _, err := manifests.Remove(ctx, name, instance.Digest.String(), &manifests.RemoveOptions{}); err != nil { 
        
           			return "", err 
        
           		} 
        
           	} 
        
           	return name, nil 
        
           }

rhatdan · 2024-05-09T12:24:53Z

Great thanks.

tnk4on · 2024-05-11T05:26:33Z

I now know why this issue slipped through the test.

Even with the current code, the first podman farm build succeeds in a clean test environment. This is because it simply pushes the generated manifest.

However, the second run requires deleting the existing manifest in the registry, which is not possible with the current code.

Along with modifying the code, the test needs to be modified so that the command is executed more than once.

$ podman run --rm -d -p 5000:5000 --name registry quay.io/libpod/registry:2.8
d8858c872ae214900da90ec55ce647ad7f8f44b2b1bd63196c9134f9a73e8528
$ podman images
REPOSITORY                TAG         IMAGE ID      CREATED       SIZE
docker.io/library/alpine  latest      ace17d5d883e  3 months ago  8.02 MB
quay.io/libpod/registry   2.8         7aff825fd082  2 years ago   23.1 MB
$ cat ../test/Containerfile 
FROM alpine
RUN arch | tee /arch.txt
RUN date | tee /built.txt
$ podman system connection ls
Name        URI                                                          Identity                    Default     ReadWrite
test-node   ssh://user@localhost:22/run/user/1000/podman/podman.sock     /home/user/.ssh/id_rsa      false       true
$ podman farm ls
Name        Connections  Default     ReadWrite
fm1         [test-node]  true        true

### first time ###

$ podman --remote farm build --tls-verify=false -t localhost:5000/test-image-5 ../test
Setting up local builder
Local builder ready
Connecting to "test-node"
Builder "test-node" ready
Farm "fm1" ready
Starting build for [{linux arm64 v8}] at "(local)"
[linux/arm64/v8@(local)] STEP 1/3: FROM alpine
[linux/arm64/v8@(local)] STEP 2/3: RUN arch | tee /arch.txt
[linux/arm64/v8@(local)] aarch64
[linux/arm64/v8@(local)] --> 171aefd5e758
[linux/arm64/v8@(local)] STEP 3/3: RUN date | tee /built.txt
[linux/arm64/v8@(local)] Sat May 11 04:53:49 UTC 2024
[linux/arm64/v8@(local)] COMMIT
[linux/arm64/v8@(local)] --> a9a944181252
[linux/arm64/v8@(local)] a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
finished build for [{linux arm64 v8}] at "(local)": built a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
Getting image source signatures
Copying blob sha256:510c42af1e324bbfc80e43c268cdfa0a381e1e8d715d8d5d5d04b7bf463b3ed1
Copying blob sha256:b09314aec293bcd9a8ee5e643539437b3846f9e5e55f79e282e5f67e3026de5e
Copying blob sha256:5e3fcf103ee086d22fed57b1e494c4fd260e7b0354e50a4e83e21185284470ae
Copying config sha256:a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
Writing manifest to image destination
Getting image list signatures
Copying 0 images generated from 1 images in list
Writing manifest list to image destination
Storing list signatures
Saved list to "localhost:5000/test-image-5"

### 2nd time ###

$ podman --remote farm build --tls-verify=false -t localhost:5000/test-image-5 ../test
Setting up local builder
Local builder ready
Connecting to "test-node"
Builder "test-node" ready
Farm "fm1" ready
Starting build for [{linux arm64 v8}] at "(local)"
[linux/arm64/v8@(local)] STEP 1/3: FROM alpine
[linux/arm64/v8@(local)] STEP 2/3: RUN arch | tee /arch.txt
[linux/arm64/v8@(local)] --> Using cache 171aefd5e7582d0143f7ce5a504df652f5e8b35b5442a99816af4e716e5a4f91
[linux/arm64/v8@(local)] --> 171aefd5e758
[linux/arm64/v8@(local)] STEP 3/3: RUN date | tee /built.txt
[linux/arm64/v8@(local)] --> Using cache a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
[linux/arm64/v8@(local)] --> a9a944181252
[linux/arm64/v8@(local)] a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
finished build for [{linux arm64 v8}] at "(local)": built a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
Getting image source signatures
Copying blob sha256:510c42af1e324bbfc80e43c268cdfa0a381e1e8d715d8d5d5d04b7bf463b3ed1
Copying blob sha256:5e3fcf103ee086d22fed57b1e494c4fd260e7b0354e50a4e83e21185284470ae
Copying blob sha256:b09314aec293bcd9a8ee5e643539437b3846f9e5e55f79e282e5f67e3026de5e
Copying config sha256:a9a9441812525f24f848fb94e66b136d5176302f6e07b12fc3ea4f38bc5c0243
Writing manifest to image destination
Error: build: error clearing list "localhost:5000/test-image-5"

tnk4on added the kind/bug Categorizes issue or PR as related to a bug. label May 8, 2024

github-actions bot added macos MacOS (OSX) related remote Problem is in podman-remote labels May 8, 2024

tnk4on mentioned this issue May 11, 2024

Fix podman-remote support for podman farm build #22673

Merged

openshift-merge-bot bot closed this as completed in #22673 May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot run `podman farm build` with podman-remote #22647

Cannot run `podman farm build` with podman-remote #22647

tnk4on commented May 8, 2024

tnk4on commented May 9, 2024

rhatdan commented May 9, 2024

tnk4on commented May 11, 2024

Cannot run podman farm build with podman-remote #22647

Cannot run podman farm build with podman-remote #22647

Comments

tnk4on commented May 8, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

tnk4on commented May 9, 2024

rhatdan commented May 9, 2024

tnk4on commented May 11, 2024

Cannot run `podman farm build` with podman-remote #22647

Cannot run `podman farm build` with podman-remote #22647