--cache-image and --publish do not work with remote docker builder

[majusmisiak]

Summary

I am running remote docker builder on TCP port (tcp://localhost:2375, no TLS).

When running pack build command with --publish and --image-cache flags I receive unauthorized error. All docker CLI commands (pull, push) work normally.

===> ANALYZING
[analyzer] ERROR: failed to initialize analyzer: validating registry read access: ensure registry read access to harbor.mydomain/cache/test:latest
ERROR: failed to build: executing lifecycle: failed with status code: 1

---

Reproduction

Steps

Create Private registry in Harbor and robot account with read/write credentials.

Create a Pod in Kubernetes:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: docker
  name: docker
spec:
  containers:
  - command:
    - /bin/cat
    env:
    - name: DOCKER_TLS_CERTDIR
      value: /certs/client
    - name: DOCKER_HOST
      value: tcp://localhost:2375
    image: docker:24
    imagePullPolicy: IfNotPresent
    name: docker
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 1Gi
    tty: true
  - command:
    - dockerd
    - -H
    - tcp://0.0.0.0:2375
    - --tls=false
    env:
    - name: DOCKER_TLS_CERTDIR
      value: /certs
    image: docker:24-dind
    imagePullPolicy: IfNotPresent
    name: dind
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 1Gi
    securityContext:
      privileged: true

Exec into docker container.

Download pack binary.

Run following commands:

docker login harbor.mydomain -u username -p password

cd /tmp
git clone https://github.com/vmware-tanzu/application-accelerator-samples.git
cd application-accelerator-samples/java-rest-service/

pack build \
    --network host \
    --docker-host inherit \
    --cache-image harbor.mydomain/cache/java-rest-service:latest \
    --publish harbor.mydomain/test/java-rest-service:latest \
    --volume "$(pwd)/bindings/certs:/platform/bindings/ca-certificates" 

OUTPUT

Warning: Using untrusted builder with volume mounts. If there is sensitive data in the volumes, this may present a security vulnerability.
latest: Pulling from paketobuildpacks/builder-jammy-base
c9e01bab5ecb: Pulling fs layer 
..........
a78e12994f88: Pull complete 
Digest: sha256:b8c59f6809d1d885c06799aaddc11f704570202fca5a019f05f61012b46b6a1b
Status: Downloaded newer image for buildpacksio/lifecycle:latest
===> ANALYZING
[analyzer] ERROR: failed to initialize analyzer: validating registry read access: ensure registry read access to harbor.mydomain/test/java-rest-service:latest
ERROR: failed to build: executing lifecycle: failed with status code: 1

See that the builder image was pulled, but the authorization to Harbor failed.

Current behavior

Authorization to registry failed.

Expected behavior

Cache image pushed to Harbor.
Build Image pushed to Harbor.

---

Environment

pack info

Pack:
  Version:  0.30.0+git-c38f7da.build-4952
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.17.0

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
(no config file found at /home/devops/.pack/config.toml)

docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7880925980b188f4c97b462f709d0db8e8962aff
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-147-generic
 Operating System: Alpine Linux v3.18 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.81GiB
 Name: jenkins-docker
 ID: de199305-cc02-45e4-88b9-6c155f9c8b19
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No swap limit support

[jjbustamante]

Hi @majusmisiak could you try to run your example without --docker-host inherit in your pack build command?, in this particular case I tried to understand the containers that are executed and I my thoughts are:

1. From the container where you are running the pack build command the docker socket is at tcp://localhost:2375 (DOCKER_HOST env) which is pointing to the other container running in the same pod
2. From the build container perspective, when using docker-host inherit indicates the socket is also running in tcp://localhost:2375 BUT from that builder container localhost is the container itself not the container running the daemon and I think this is breaking the hostname resolution