Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nerdctl pull from private registry mirror is missing Authorization header #2844

Open
networkhermit opened this issue Feb 27, 2024 · 0 comments
Open

nerdctl pull from private registry mirror is missing Authorization header #2844

networkhermit opened this issue Feb 27, 2024 · 0 comments
Labels
area/login authentification/ login bug Something isn't working

Comments

@networkhermit
Copy link

networkhermit commented Feb 27, 2024 •
edited

Description

I tried to use nerdctl with registry mirrors served by private harbor, but I got 401 Unauthorized error.

After some investigation, I found that when using registry mirrors, nerdctl doesn't pass the Authorization: Basic xxxx header to /service/token endpoints, causing the final 401 Unauthorized error.

Steps to reproduce the issue

  1. configure containerd to use registry mirrors:

/etc/containerd/config.toml

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

/etc/containerd/certs.d/gcr.io/hosts.toml

server = "https://gcr.io"

[host."https://harbor.example.com:8443/v2/gcr.io"]
  capabilities = ["pull", "resolve"]
  override_path = true

  1. use nerdctl login to harbor.example.com:8443, optionally setup mitmproxy to view the api traffic

  2. directly pull the image from harbor is successfully

❯ sudo HTTPS_PROXY=http://127.0.0.1:8080 nerdctl --debug-full pull harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 0.3 s                                             total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 14:05:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=7bc1d436bafba2a71e19ec32cfcc06b8; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=7 response.header.x-request-id=cb37ff75-4c43-4d62-a565-2d767adbd4d8 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.inde
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 2.0 s                                             total:   0.0 B (0.0 B/s)
DEBU[0002] fetch response received                       host="harbor.example.com:8443" response.header.content-length=1163 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 27 Feb 2024 14:05:22 GMT" response.header.docker-content-digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8\"" response.header.server=istio-envoy response.header.set-cookie="sid=1fbb1be72a6e2fd04a4962cfd6997843; Path=/; HttpOnly" response.header.x-envoy-upstream-service-time=1660 response.header.x-request-id=6dfd0fa9-f711-4512-a26f-ed627a26ab73 response.status="200 OK" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0002] resolved                                      desc.digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" host="harbor.example.com:8443"
DEBU[0002] fetch                                         digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1163
DEBU[0002] fetch                                         digest="sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc" mediatype=application/vnd.docker.container.image.v1+json size=4843
DEBU[0002] layer unpacked                                duration="221.474µs" layer="sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964"
DEBU[0002] layer unpacked                                duration="87.424µs" layer="sha256:2d4828968d6ca90e549ac054b8330c28081e02b31574f663fe8dae9c03275103"
DEBU[0002] layer unpacked                                duration="62.677µs" layer="sha256:b28a9b13dc6dccb175978a3e34f0ee5ec435e1328b9ded306724e490388585e4"
DEBU[0002] layer unpacked                                duration="92.493µs" layer="sha256:031224e6222c780a9783d9d287f6c25e37f16c800ba4bbb13e7f9c77036fb811"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest:                        resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.2 s                                                                    total:   0.0 B (0.0 B/s)
  1. pulling the same image from registry mirrors fails as 401
❯ sudo HTTPS_PROXY=127.0.0.1:8080 nerdctl --debug-full pull gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/gcr.io
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=63cc905e0a57374c5292b8273645d06c; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=3 response.header.x-request-id=d0e29292-1e9e-4f15-ab65-51d37f1dd851 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=198 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=5ce41f88f2ba4e9246430cfa8e46312f; Path=/; HttpOnly" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-envoy-upstream-service-time=1 response.header.x-request-id=0c1dc612-bbaa-4638-ba67-18f0a69006cf response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Basic realm=\"harbor\"" host="harbor.example.com:8443"
DEBU[0000] resolving                                     host=gcr.io
DEBU[0000] do request                                    host=gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://gcr.io/v2/cadvisor/cadvisor/manifests/latest"
gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 133.1s                  total:   0.0 B (0.0 B/s)
INFO[0133] trying next host                              error="failed to do request: Head \"https://gcr.io/v2/cadvisor/cadvisor/manifests/latest\": Bad Gateway" host=gcr.io
FATA[0133] failed to resolve reference "gcr.io/cadvisor/cadvisor:latest": unexpected status from HEAD request to https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io: 401 Unauthorized
  1. Browse through the mitmproxy api requests, I can see the request tohttps://harbor.example.com:8443/service/token?scope=repository%3Acadvisor%2Fcadvisor%3Apull&scope=repository%3Agcr.io%2Fcadvisor%2Fcadvisor%3Apull&service=harbor-registry lacks the Authorization: Basic xxxx header, hence results the final 401 error.

I also checked the mitmproxy api traffic using ctr and crictl, on successfully image pulling from registry mirrors, the above Authorization: Basic xxxx header is both present.

Describe the results you received and expected

nerdctl pull from private registry mirror works, the credential of the mirror registry can be passed and reused from nerdctl login credential and hence adding the missing Authorization: Basic xxxx header.

What version of nerdctl are you using?

Client:
Version: 1.7.2
OS/Arch: linux/amd64
Git commit: e32c4b0
buildctl:
Version:

Server:
containerd:
Version: v1.7.13
GitCommit: 7c3aca7a610df76212171d200ca3811ff6096eb8.m
runc:
Version: 1.1.12

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: v1.7.13
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.7.6-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125GiB
Name: arch
ID: 36b0778a-4ce8-4dca-af98-5afa21440138
@networkhermit networkhermit added the kind/unconfirmed-bug-claim Unconfirmed bug claim label Feb 27, 2024
@AkihiroSuda AkihiroSuda added bug Something isn't working area/login authentification/ login and removed kind/unconfirmed-bug-claim Unconfirmed bug claim labels Feb 27, 2024
minuk-dev added a commit to minuk-dev/containerd that referenced this issue Apr 28, 2024
@minuk-dev
Add HostsFromHostDir() to remotes/docker/config
652da9f 
`remotes/docker/config` package provides `ConfigureHosts()` and
`HostDirFromRoot()` functions only. But, its caller cannot know the
actual hosts from a host. It causes that the caller cannot call
`ConfigureHosts()` method with its proper Credentials.
e.g. containerd/nerdctl#2844 - "nerdctl pull from private registry
mirror is missing Authorization header"

Signed-off-by: Min Uk Lee <minuk.dev@gmail.com>
@minuk-dev minuk-dev mentioned this issue Apr 28, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/login authentification/ login bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AkihiroSuda @networkhermit