New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
INFO[0101] failed to verify token: token signed by untrusted key with ID: #4299
Comments
Hmm.. it looks like a workaround might be to have the auth service not send a key id, or to have the registry not validate it, but is that a good solution? |
I feel like your trusted keys are not being loaded properly or you're not loading them at all: distribution/registry/auth/token/accesscontroller.go Lines 282 to 287 in d9815da
You need to provide a path to your JWKS and pass it to the config file via the distribution/registry/auth/token/accesscontroller.go Lines 265 to 270 in d9815da
You can of course "just" rely on the Cert bundle path to which is also passed via a config param: distribution/registry/auth/token/accesscontroller.go Lines 258 to 263 in d9815da
|
@milosgajdos I tried that and if the key is not present in the registry config it results in a panic - "panic: unable to configure authorization (token): token auth requires at least one token signing key". Likewise on the auth server I can see that the KeyID changes if I update the key but remains stable if I don't, so I'm pretty confident it is loading the file. I tried not sending the keyID and that resulted in the message "INFO[0050] failed to verify token: go-jose/go-jose: no x5c header present in message". I am wondering if the format of the KeyID field changed and docker-registry can no longer handle values produced by a different method. |
This seems to be the breaking change: #4096 Docker-registry can no longer process the kid values that were produced by the docker-libtrust library. It's not clear if its an issue of making the authentication service produce kid values the registry does understand, adding some compatibility code to honor the format that docker-libtrust used (the presence of colons seems to to be an indicator that its a docker-libtrust style id, or if there are other changes required such as the x5c field referenced above. |
Confirmed that if I run https://github.com/milosgajdos/distribution/tree/1d410148efe6d1b7fd56457507a9dd465b105ec4 instead, authentication works again. |
This is the correct behaviour -- how do you want to verify the token if you don't provide the trusted keys to registry?
FYI: we have deprecated The format of the auth0 have a nice breakdown of what the JWKs look like: https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-set-properties We load the JWKs we find in the file specified via the config: distribution/registry/auth/token/accesscontroller.go Lines 282 to 287 in d9815da
If you don't send the keyID and load the JWKs then the verification should correctly fail - how is the registry to know what key to verify if it doesn't have the keyID to look it up in its trusted keys? The error you are seeing happens when you rely on TLS certs but again fail to get registry to load them so it can use them for verification. |
Description@milosgajdos we're encountering the same problem while testing 3.0.0-alpha.1: We run an auth_server and a registry from the following compose configuration.
Logsauth_server logs:
registry logs:
ReproduceTo reproduce, run the following configuration locally: File Structure
certificate.pem and private.keyGenerate SSL keypair using:
docker-compose.yml
auth-server-config.yml
Reproduce Error
|
Can you please format your message properly? It's impossible to parse useful info from it, I'm afraid. Also, it's hard to investigate in general if I can't see the JWT your auth server is issuing. |
@milosgajdos I'm very sorry for my cryptic comment. I just updated it with a detailed explanation of the error and a sample configuration to reproduce it locally. Thanks for your great work maintaining this essential project! |
Thanks, that's useful. Off the top of my head, I can see that your JWT contains the optional
If you are going to include the Are you signing these keys with TLS certs? Now I see where all these woes are coming from...The original code was checking the certs chain first and if that passed the validation, we'd just return; the current code checks the JWTs first and certs later, which surfaced a lot of issues to folks. I've opened a PR in |
Thanks for your quick help! I noticed earlier today that Please excuse my foolish question: What exactly is the "trusted keys" config and where can I specify the registry to trust a certain key? We indeed use TLS certificates to generate tokens. In production, the registry is deployed behind a load-balancer that handles TLS for all services. Therefore, no |
Yeah we're missing
Basically, it lets you specify the path to the keys you can use for verifying the JWT tokens -- in the similar way as the cert bundle path is specified. If you are using TLS certs to sign your tokens, then once my PR is merged into |
Thanks a lot! I also just opened #4345 with a minimal proposal to include the new parameter in the docs. I can also write a more detailed description if desired. |
Description
I'm testing updating to a recent build of docker-registry but I'm getting stuck on the authorization part. The error in the file is:
INFO[0101] failed to verify token: token signed by untrusted key with ID: "xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx"
The auth service is using docker/libtrust to generate the keyid:
https://github.com/cesanta/docker_auth/blob/38e7252690dc31ab7ccd4185a2dead973099f37c/auth_server/server/server.go#L384
https://github.com/cesanta/docker_auth/blob/38e7252690dc31ab7ccd4185a2dead973099f37c/auth_server/server/config.go#L88
So it seems to be docker-registry itself making the determination that the key is untrusted. However it doesn't generate any errors or warnings when loading the key, only when processing the auth response. The error message doesn't provide any hints as to why it is untrusted, and there are no other relevant messages in the log. This last worked using a build of docker-registry from early 2022 (commit c202b9b), the only change is using a March 2024 build (commit 663b430) - though I also tried a newer certificate to see if it gave a different result.
Reproduce
Setting up a local registry can be complex, I'll wait to see if its something simple like docker-registry only supports rsa-4096, or has banned values for common name, etc.
Expected behavior
The server should accept the positive response from the auth server and allow access.
registry version
/usr/bin/docker-registry github.com/distribution/distribution/v3 v3.0.0-alpha.1.m+unknown
Additional Info
No response
The text was updated successfully, but these errors were encountered: