pack builder create cannot authenticate against Azure Container Registry (ACR)

[tgquan67]

Summary

I have a private docker repository on ACR, which I can login to, pull from and push to as normal. My run-image and build-image are also stored there.
The builder is named private-registry.azurecr.io/builder/test:latest with the following config:

# Buildpacks to include in builder
[[buildpacks]]
id = "test-buildpack"
uri = "../buildpacks/test-buildpack"

# Order used for detection
[[order]]
    # This buildpack will display build-time information (as a dependency)
    [[order.group]]
    id = "test-buildpack"

# Stack that will be used by the builder
[stack]
id = "com.example.buildpacks.stacks.test"
# This image is used at runtime
run-image = "private-registry.azurecr.io/images/test-run-image:latest"
# This image is used at build-time
build-image = "private-registry.azurecr.io/images/test-build-image:latest"

EDIT: I need to clarify that in this case I logged in using a pair of service principal ID/secret with enough permission.

---

Reproduction

Steps

1. Run pack builder create private-registry.azurecr.io/builder/test:latest --config builder.toml

Current behavior

ERROR: invalid run image config: failed to fetch image: Error response from daemon: Head "https://private-registry.azurecr.io/v2/images/test-run-image:/manifests/latest": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

Expected behavior

Builder should be created without any problem.

---

Environment

pack info

Pack:
  Version:  0.33.1
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.18.4

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
(no config file found at /home/me/.pack/config.toml)

docker info

Client: Docker Engine - Community
 Version:    24.0.9
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 34
 Server Version: 24.0.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-171-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 62.83GiB
 Name: quan-work-ubuntu
 ID: DPTV:QXJW:TN6C:CVLT:2DU7:DGNV:7IRF:UGNG:AOPX:T3VE:JS3X:6KIP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:32000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[natalieparellano]

@tgquan67 could this be the same as #2064 in that client certs are needed (though the pack command is different)?

It's worth noting, the pack client constructs its keychain here - since pack is optimized for local development, we assume the default keychain is sufficient but perhaps we need something more like the lifecycle keychain which can make use of IaaS credential helpers.

[tgquan67]

@natalieparellano no, these 2 problems are not related, since for normal daily usage, the registry on ACR doesn't require any client certificate.
Edit: it seems you misunderstood my case in both ticket. The cert here is for the server to verify client's identity, not the root cert in case the server is self-signed.

[natalieparellano]

since for normal daily usage, the registry on ACR doesn't require any client certificate.

In that case, I suspect the problem is with the default keychain not having the right credentials

perhaps we need something more like the lifecycle keychain which can make use of IaaS credential helpers.

I could be wrong, but maybe this is what we need. Do you have any opinion here?

[tgquan67]

Okay, here is an update. When I login with az acr login -n private-registry instead of logging in directly using a service principal, it worked properly. The ~/.docker/config.json now looks like this

{
        "auths": {
                "private-registry.azurecr.io": {
                        "auth": "MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAwOg==",
                        "identitytoken": "<redacted>"
                }
        }
}

It seems a credential helper was used, but I don't see it specified in this file.