OCI with private Azure Container Registry: download of policies fails with HTTP 403

[rbinder-vg]

Short description

I want to use leverage OPAs OCI functionality to retrieve policies from a private Azure Container Registry.
There is a error (403 Server failed to authenticate the request.) when downloading the blobs.
The download of the manifest seems ok.

If i use the the oras cli with the same service principal the download works as expected.

• OPA version 0.60.0
• logs:

{"level": "debug","msg": "OCI - Download starting.","time": "2024-02-13T10:06:46Z"}
{"level": "debug","msg": "OCIDownloader: using auth plugin: *rest.bearerAuthPlugin","time": "2024-02-13T10:06:46Z"}
{"host": "foo.azurecr.io","level": "debug","msg": "resolving","time": "2024-02-13T10:06:46Z"}
{"host": "foo.azurecr.io","level": "debug","msg": "do request","request.header.accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent": "containerd/1.7.11+unknown","request.method": "HEAD","time": "2024-02-13T10:06:46Z","url": "https://foo.azurecr.io/v2/bar/opa/manifests/0.0.1"}
{"headers": {"Content-Type": ["application/json"],"User-Agent": ["Open Policy Agent/0.60.0 (linux, amd64)"]},"level": "debug","method": "POST","msg": "Sending request.","time": "2024-02-13T10:06:46Z","url": "https://telemetry.openpolicyagent.org/v1/version"}
{"host": "foo.azurecr.io","level": "debug","msg": "fetch response received","response.header.access-control-expose-headers": "Docker-Content-Digest","response.header.access-control-expose-headers.1": "WWW-Authenticate","response.header.access-control-expose-headers.2": "Link","response.header.access-control-expose-headers.3": "X-Ms-Correlation-Request-Id","response.header.content-length": "540","response.header.content-type": "application/vnd.oci.image.manifest.v1+json","response.header.date": "Tue, 13 Feb 2024 10:06:46 GMT","response.header.docker-content-digest": "sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4","response.header.docker-distribution-api-version": "registry/2.0","response.header.etag": "\"sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4\"","response.header.server": "openresty","response.header.strict-transport-security": "max-age=31536000; includeSubDomains","response.header.strict-transport-security.1": "max-age=31536000; includeSubDomains","response.header.x-content-type-options": "nosniff","response.header.x-ms-client-request-id": "","response.header.x-ms-correlation-request-id": "b27bfa15-7a0f-4b25-9340-08b695a060b4","response.header.x-ms-request-id": "973911b1-4e80-45da-955e-c59ca3f97d48","response.status": "200 OK","time": "2024-02-13T10:06:46Z","url": "https://foo.azurecr.io/v2/bar/opa/manifests/0.0.1"}
{"desc.digest": "sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4","host": "foo.azurecr.io","level": "debug","msg": "resolved","time": "2024-02-13T10:06:46Z"}
{"digest": "sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4","level": "debug","msg": "do request","request.header.accept": "application/vnd.oci.image.manifest.v1+json, */*","request.header.user-agent": "containerd/1.7.11+unknown","request.method": "GET","time": "2024-02-13T10:06:46Z","url": "https://foo.azurecr.io/v2/bar/opa/manifests/sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4"}
{"digest": "sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4","level": "debug","msg": "fetch response received","response.header.access-control-expose-headers": "Docker-Content-Digest","response.header.access-control-expose-headers.1": "WWW-Authenticate","response.header.access-control-expose-headers.2": "Link","response.header.access-control-expose-headers.3": "X-Ms-Correlation-Request-Id","response.header.connection": "keep-alive","response.header.content-length": "540","response.header.content-type": "application/vnd.oci.image.manifest.v1+json","response.header.date": "Tue, 13 Feb 2024 10:06:47 GMT","response.header.docker-content-digest": "sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4","response.header.docker-distribution-api-version": "registry/2.0","response.header.etag": "\"sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4\"","response.header.server": "openresty","response.header.strict-transport-security": "max-age=31536000; includeSubDomains","response.header.strict-transport-security.1": "max-age=31536000; includeSubDomains","response.header.x-content-type-options": "nosniff","response.header.x-ms-client-request-id": "","response.header.x-ms-correlation-request-id": "aed98589-403e-45b1-959a-5885d398990c","response.header.x-ms-request-id": "bc0b805f-46ec-40dd-9363-580355a4af17","response.status": "200 OK","time": "2024-02-13T10:06:47Z","url": "https://foo.azurecr.io/v2/bar/opa/manifests/sha256:6c41e52daf965ba12ebbaf0f78a9ab0b9197430bee5099d5b11ec48d032a0ea4"}
{"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a","level": "debug","msg": "do request","request.header.accept": "application/vnd.oci.image.config.v1+json, */*","request.header.user-agent": "containerd/1.7.11+unknown","request.method": "GET","time": "2024-02-13T10:06:47Z","url": "https://foo.azurecr.io/v2/bar/opa/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"}
{"digest": "sha256:f9728c7e8e8a3d94b3c84811c2f7e398507786742b954d791f3d231b65daaf2e","level": "debug","msg": "do request","request.header.accept": "application/vnd.oci.image.layer.v1.tar+gzip, */*","request.header.user-agent": "containerd/1.7.11+unknown","request.method": "GET","time": "2024-02-13T10:06:47Z","url": "https://foo.azurecr.io/v2/bar/opa/blobs/sha256:f9728c7e8e8a3d94b3c84811c2f7e398507786742b954d791f3d231b65daaf2e"}
{"digest": "sha256:f9728c7e8e8a3d94b3c84811c2f7e398507786742b954d791f3d231b65daaf2e","level": "debug","msg": "fetch response received","response.header.content-length": "321","response.header.content-type": "application/xml","response.header.date": "Tue, 13 Feb 2024 10:06:46 GMT","response.header.server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0","response.header.x-ms-request-id": "4a3e2ef9-501e-0020-4c64-5e82df000000","response.status": "403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.","time": "2024-02-13T10:06:47Z","url": "https://foo.azurecr.io/v2/bar/opa/blobs/sha256:f9728c7e8e8a3d94b3c84811c2f7e398507786742b954d791f3d231b65daaf2e"}
{"level": "error","msg": "Bundle load failed: failed to pull foo.azurecr.io/bar/opa:0.0.1: download for 'foo.azurecr.io/bar/opa:0.0.1' failed: failed to ingest: copy failed: httpReadSeeker: failed open: unexpected status code https://foo.azurecr.io/v2/bar/opa/blobs/sha256:f9728c7e8e8a3d94b3c84811c2f7e398507786742b954d791f3d231b65daaf2e: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.","name": "authz","plugin": "bundle","time": "2024-02-13T10:06:47Z"}

Steps To Reproduce

configuration.yml:

services:
  acr-registry:
    url: https://foo.azurecr.io
    type: oci
    credentials:
      bearer:
        scheme: "Basic"
        token: "<user>:<secret>"
bundles:
  authz:
    service: acr-registry
    resource: foo.azurecr.io/bar/opa:0.0.1
    persist: true
    polling:
      min_delay_seconds: 30
      max_delay_seconds: 120

Start server with docker:

docker run --rm --volume "$(pwd):/wd" --workdir /wd -ti --entrypoint opa openpolicyagent/opa:0.60.0 run -c configuration.yml

Expected behavior

Opa policies successfully loaded.

Additional context

[ashutosh-narkar]

@rbinder-vg thanks for reporting this. Are you able to repro this even with the latest OPA release?

@carabasdaniel @gitu any ideas what's happening here?

[rbinder-vg]

@rbinder-vg thanks for reporting this. Are you able to repro this even with the latest OPA release?

@carabasdaniel @gitu any ideas what's happening here?

@ashutosh-narkar same issue with latest release and with main.

[carabasdaniel]

Hi @rbinder-vg,

If you try a very basic docker pull using those credentials does that work ?

[rbinder-vg]

Hi @rbinder-vg,

If you try a very basic docker pull using those credentials does that work ?

Hi @carabasdaniel,

I made a oras pull and it worked fine.

[rbinder-vg]

Hi @carabasdaniel @ashutosh-narkar , i found the cause for the bug: the ACR sends a redirect to an azure blob storage with a SAS token as query parameter and OPA is setting the Authorization header at the same time. This results in the strange HTTP 403 error.

OPA:

ORAS ( with oras pull):

[carabasdaniel]

Hi @rbinder-vg,

This really seems linked to #6580 as it's the same behavior. I think this might be related to how the plugin authorizer works in the docker resolver here: https://github.com/open-policy-agent/opa/blob/main/download/oci_download.go#L346

Can you try to use a custom plugin as a work-around ?

[rbinder-vg]

Hi @rbinder-vg,

This really seems linked to #6580 as it's the same behavior. I think this might be related to how the plugin authorizer works in the docker resolver here: https://github.com/open-policy-agent/opa/blob/main/download/oci_download.go#L346

Can you try to use a custom plugin as a work-around ?

A little bit hacky, but this is working:

func (p *Plugin) Prepare(req *http.Request) error {
	if !strings.Contains(req.URL.Host,"azurecr.io") {
		return nil
	}

	req.Header.Add("Authorization", p.config.Authz)
	return nil
}

[carabasdaniel]

I think there is an Azure and an AWS plugin for the REST client. With the latest changes in the OCI downloader the docker resolver gets a client from the plugins if I remember correctly (auth plugin used by default). @ashutosh-narkar would those plugins be usable for the OCI downloader as well ?

[ashutosh-narkar]

would those plugins be usable for the OCI downloader as well ?

Looking at this change, only the Azure one seems unavailable atm.

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.