pack uses local computer for DNS resolution of image registry name during RESTORER phase

[yters]

Summary

When using pack to build an app, the RESTORER phase attempts to use the local DNS to resolve the image registry.
This is taking place within a VPN, incase that is relevant.

$ pack build hello-app --path . --builder <path>/builder-python3.11:debug                                         
...         
===> RESTORING                                                                                                               
[restorer] ERROR: failed to pull run image: failed to initialize remote image: connect to repo store "<domain>/builder-python3.11:debug": Get "https://<domain>/v2/": dial tcp: lookup <domain> on 192.168.1.1:53: no such host
ERROR: failed to build: executing lifecycle: failed with status code: 1

---

Reproduction

Steps

1. Create a build image with the following Dockerfile named <path>/build-base-python3.11:debug.

FROM registry.access.redhat.com/ubi8-minimal:8.8

ARG STACK_ID="debug"

ENV CNB_USER_ID=${CNB_UID:-1000}
ENV CNB_GROUP_ID=${CNB_GID:-1000}

RUN microdnf update
RUN microdnf install python3.11

RUN groupadd cnb --gid ${CNB_GROUP_ID} 
RUN useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb 

USER cnb
LABEL io.buildpacks.stack.id=${STACK_ID}
ENV CNB_STACK_ID=${STACK_ID}
ENV CNB_USER_ID=${CNB_UID:-1000}
ENV CNB_GROUP_ID=${CNB_GID:-1000}

2. Create a run image with the following Dockerfile named <path>/run-base:debug

FROM registry.access.redhat.com/ubi8-minimal:8.8

ARG STACK_ID="debug"

ENV APP_USER_ID=${APP_UID:-31460}
ENV APP_GROUP_ID=${APP_GID:-31460}

LABEL io.buildpacks.stack.id=${STACK_ID}
ENV CNB_STACK_ID=${STACK_ID}

4. Create an extension with the following files, named <path>/extension-dnf-install:debug.
   extension.toml

api = "0.10"

[extension]
  id = "dnf-install"
  name = "dnf python3.11 RPM installer"
  description = "Extension that installs python3.11 RPM in the run image."
  version = "debug"

[[targets]]
os = "linux/amd64"
arch = "amd64"

generate/run.Dockerfile

ARG base_image
FROM ${base_image}

USER root

RUN microdnf install python3.11

ARG user_id
USER \${user_id}

5. Create a buildpack from the hello-word example buildpack, named <path>/buildpack-hello-world:debug.
6. Create a builder with the following builder.toml, named <path>/builder-python3.11:debug.

[[extensions]]
uri = "<path>/extension-dnf-install:debug"
id = "dnf-install"

[[order-extensions]]
[[order-extensions.group]]
id = "dnf-install"
version = "debug"

[[buildpacks]]
uri = "<path>/buildpack-hello-world:debug"
id  = "hello-world"

[[order]]
[[order.group]]
id = "hello-world"
version = "debug"

[stack]
id = "debug"
run-image = "<path>/run-base:debug"
build-image = "<path>/build-base-python3.11:debug"

7. Use pack build command with builder-python3.11:debug image to create an app. Create the following files in the current directory.
   config/run.toml

[[images]]
 image = "<path>/run-base:debug"

source/hello.sh

#!/bin/sh

echo hello

Then, run the following command to build the app.

$ pack build hello-app --path . --builder <path>/builder-python3.11:debug

You'll see a result like this.

$ pack build hello-app --path . --builder <path>/builder-python3.11:debug                                         
...         
===> RESTORING                                                                                                               
[restorer] ERROR: failed to pull run image: failed to initialize remote image: connect to repo store "<domain>/builder-python3.11:debug": Get "https://<domain>/v2/": dial tcp: lookup <domain> on 192.168.1.1:53: no such host
ERROR: failed to build: executing lifecycle: failed with status code: 1

Current behavior

In the RESTORER phase it will attempt to use the local computer to DNS resolve the image registry domain:

===> RESTORING                                                                                                               
[restorer] ERROR: failed to pull run image: failed to initialize remote image: connect to repo store "<domain>/builder-python3.11:debug": Get "https://<domain>/v2/": dial tcp: lookup <domain> on 192.168.1.1:53: no such host
ERROR: failed to build: executing lifecycle: failed with status code: 1

Expected behavior

With builders created using an older version of pack DNS resolution worked correctly.

Context

lifecycle version

Lifecycle version is 0.17.2.

platform version(s)

$ pack report
Pack:
  Version:  0.32.1+git-b14250b.build-5241
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.17.2

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
  experimental = true
  lifecycle-image = "buildpacksio/lifecycle:0.17.1"
  layout-repo-dir = "/home/<redacted>/.pack/layout-repo"

$ docker info
Client: Docker Engine - Community
 Version:    25.0.1
 Context:    default
 Debug Mode: false                                            
 Plugins:                                                     
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
                                                              
Server:           
 Containers: 35     
  Running: 1         
  Paused: 0  
  Stopped: 34               
 Images: 51
 Server Version: 25.0.1                          
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.13-100.fc38.x86_64
 Operating System: Fedora Linux 38 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.03GiB
 Name: fedora
 ID: 6a61b820-9dc5-4e48-9607-b2df128276e9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[natalieparellano]

Hi yters - thank you for this. Can you try running your pack build command with --network host?

With builders created using an older version of pack DNS resolution worked correctly.

Could you also share your lifecycle version? And are you using extensions? Newer versions of the lifecycle will try to pull the run image during restore (IF extensions switched the run image) so that target information can be populated and/or the manifest can be downloaded for use by the extender. The reference provided by extensions (recorded in analyzed.toml) should be reachable from the build container.