Unable to use letsencrypt certification generation

[creekorful]

Trying to use the registry with Letsencrypt builtin automatic certification generation fails with following error:

Error 403 - urn:acme:error:unauthorized - Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2

As Letsencrypt as announced here the account creation using API v1 is impossible since November 2019. It would be good to refactor registry to use the new ACMEv2 API to create account.

[codeclown]

In the meantime, anyone who needs to start a registry manually with Lets Encrypt... click Details:

Run certbot in standalone mode to generate certificates:

$ sudo certbot certonly --standalone

...

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem

...

Files inside /etc/letsencrypt/live/example.com are symlinks! So you can't just mount the domain-specific folder when starting the container.

Instead you need to mount the whole letsencrypt-folder in order to keep the symlink paths intact. Relevant options for docker run:

  -e REGISTRY_HTTP_ADDR="0.0.0.0:443" \
  -v /etc/letsencrypt:/etc/letsencrypt \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/letsencrypt/live/example.com/fullchain.pem \
  -e REGISTRY_HTTP_TLS_KEY=/etc/letsencrypt/live/example.com/privkey.pem \
  registry:2

Note that, as usual with certbot standalone, you should also remember to add a cronjob to run certbot renew automatically.

[cameron]

Five months and not so much as peep re a documented feature that simply doesn't work as advertised? Boooo.

[nikandlv]

Got the same issue here

msg="register: acme: Error 403 - urn:acme:error:unauthorized - Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details." 

[wilminator]

Edit- the shell command was bad. Changed from bash to sh. Also updated cp command to keep executable bit set.

I was able to get this to work on June 25, 2020. But I had to recompile the registry binary and update the image with it.

• To recompile the registry executable, make a directory, change to it, and run this:

docker run --rm -v $(pwd):/mnt -it golang:alpine /bin/sh -c "apk update; apk upgrade; apk add --no-cache bash git openssh;go get github.com/docker/distribution/cmd/registry; mkdir /mnt/bin; cp -pv /go/bin/registry /mnt/bin"

• To build a registry image that works, use this Dockerfile from the directory you created:

FROM registry:latest
COPY ./bin/registry /bin/registry

[wilminator]

For the devs, the probable quick fix here is to rebuild your 1.8 image with current golang libraries; I think that the library that enables Let's Encrypt has recently been updated to use the ACME 2.0 protocol and we just need to recompile against it.

[dorgan]

I was able to get this to work on June 25, 2020. But I had to recompile the registry binary and update the image with it.

• To recompile the registry executable, make a directory, change to it, and run this:

mkdir fixme
cd fixme
docker run --rm -v $(pwd):/mnt -it golang:alpine /bin/bash -c "apk update; apk upgrade; apk add --no-cache bash git openssh;go get github.com/docker/distribution/cmd/registry; mkdir /mnt/bin; cp -v /go/bin/registry /mnt/bin"

• To build a registry image that works, use this Dockerfile from the directory you created:

FROM registry:latest
COPY ./bin/registry /bin/registry

Trying to run that i get:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "/bin/bash": stat /bin/bash: no such file or directory": unknown.

[robogeek]

Ditto

[wilminator]

@dorgan and @robogeek, I goofed the shell to use. I updated the docker command; it now uses /bin/sh, not /bin/bash. I verified that it works on Ubuntu 2004 on Windows, so it should behave itself on any *nix system now. You also don't need the additional mkdir and cd statements, as long as you don't mind a ./bin directory in your current dir when running the docker command.

[robogeek]

@wilminator I did not get the /bin/bash error. Instead I got the error at the top about ACME v1.

[wilminator]

@robogeek, I want to know if you tried using the stock registry image or if you tried my fix. Thanks.

[robogeek]

I used registry:latest

[locinus]

To complete @wilminator answer: the docker container was unable to reach the DNS in my setting, so I added the --network host option, and it worked like a charm.

• To recompile the registry executable, make a directory, change to it, and run this:

docker run --rm -v $(pwd):/mnt --network host -it golang:alpine /bin/sh -c "apk update; apk upgrade; apk add --no-cache bash git openssh;go get github.com/docker/distribution/cmd/registry; mkdir /mnt/bin; cp -pv /go/bin/registry /mnt/bin"

• To build a registry image that works, use this Dockerfile from the directory you created:

FROM registry:latest
COPY ./bin/registry /bin/registry

• I then ran

docker build --tag registry:fixed .

and used the registry:fixed image.

[luc122c]

Hello, I've just been affected by this. As discussed above, LetsEncrypt are deprecating ACMEv1. As part of the process they're running brown-outs. I just so happened to be setting up a private docker repository when they started one.

Come on Docker! Get upgraded to ACMEv2 already!

FATA[0000] create client: get directory at 'https://acme-v01.api.letsencrypt.org/directory': acme: Error 403 - urn:acme:error:serverInternal - ACMEv1 Brownout in Progress. ACMEv1 will fully turn off on June 1, 2021. Check https://letsencrypt.status.io/ for more details.

[oglematt]

Trying to implement the answer from @wilminator and I'm getting the following output. I know next to nothing about golang, so I'm not sure where to even begin. Any help would be greatly appreciated.

go: downloading github.com/docker/distribution v2.7.1+incompatible
go: downloading github.com/Azure/azure-sdk-for-go v0.2.0-beta
go: downloading github.com/Azure/azure-sdk-for-go v52.3.1+incompatible
go: downloading github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d
go: downloading github.com/aws/aws-sdk-go v1.37.28
go: downloading github.com/bshuster-repo/logrus-logstash-hook v1.0.0
go: downloading github.com/bugsnag/bugsnag-go v1.9.0
go: downloading github.com/bugsnag/bugsnag-go v2.1.0+incompatible
go: downloading github.com/denverdino/aliyungo v0.0.0-20210222084345-ddfe3452f5e8
go: downloading github.com/docker/go-metrics v0.0.1
go: downloading github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7
go: downloading github.com/garyburd/redigo v1.6.2
go: downloading github.com/gorilla/handlers v1.5.1
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/mitchellh/mapstructure v1.4.1
go: downloading github.com/ncw/swift v1.0.53
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/opencontainers/image-spec v1.0.1
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/yvasiyarov/gorelic v0.0.7
go: downloading golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
go: downloading golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93
go: downloading google.golang.org/api v0.41.0
go: downloading google.golang.org/cloud v0.79.0
go: downloading google.golang.org/cloud/storage v1.14.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading rsc.io/letsencrypt v0.0.3
go get: google.golang.org/cloud@none updating to
        google.golang.org/cloud@v0.79.0: parsing go.mod:
        module declares its path as: cloud.google.com/go
                but was required as: google.golang.org/cloud
cp: can't stat '/go/bin/registry': No such file or directory

[wilminator]

Trying to implement the answer from @wilminator and I'm getting the following output. I know next to nothing about golang, so I'm not sure where to even begin. Any help would be greatly appreciated.

go: downloading github.com/docker/distribution v2.7.1+incompatible
go: downloading github.com/Azure/azure-sdk-for-go v0.2.0-beta
go: downloading github.com/Azure/azure-sdk-for-go v52.3.1+incompatible
go: downloading github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d
go: downloading github.com/aws/aws-sdk-go v1.37.28
go: downloading github.com/bshuster-repo/logrus-logstash-hook v1.0.0
go: downloading github.com/bugsnag/bugsnag-go v1.9.0
go: downloading github.com/bugsnag/bugsnag-go v2.1.0+incompatible
go: downloading github.com/denverdino/aliyungo v0.0.0-20210222084345-ddfe3452f5e8
go: downloading github.com/docker/go-metrics v0.0.1
go: downloading github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7
go: downloading github.com/garyburd/redigo v1.6.2
go: downloading github.com/gorilla/handlers v1.5.1
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/mitchellh/mapstructure v1.4.1
go: downloading github.com/ncw/swift v1.0.53
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/opencontainers/image-spec v1.0.1
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/yvasiyarov/gorelic v0.0.7
go: downloading golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
go: downloading golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93
go: downloading google.golang.org/api v0.41.0
go: downloading google.golang.org/cloud v0.79.0
go: downloading google.golang.org/cloud/storage v1.14.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading rsc.io/letsencrypt v0.0.3
go get: google.golang.org/cloud@none updating to
        google.golang.org/cloud@v0.79.0: parsing go.mod:
        module declares its path as: cloud.google.com/go
                but was required as: google.golang.org/cloud
cp: can't stat '/go/bin/registry': No such file or directory

@oglematt Truth be told, I'm not a golang guy either; I just make the computers work ;)

It looks like an upstream Google cloud library was updated 4 hours before your post google-cloud-go and they renamed the library. Since the go.mod configuration file for registry still uses the old name, go panics because it downloaded a module that has a different name than what it was expecting.

The devs for this project probably need to fix their requirements file to use the new name of the Google Cloud module. I have submitted a ticket #3377, and if I have time will try to offer a patch to fix it.

[wilminator]

Hello,
With some help from my issue respondents (@milosgajdos and @wy65701436), I was able to get a working registry executable built that supports ACME v2.

docker run --rm -v "C:\Users\me\docreg:/mnt" -it golang:alpine /bin/sh -c "apk update&& apk upgrade&& apk add --no-cache bash git openssh make&& cd /opt&& git clone https://github.com/docker/distribution&&cd /opt/distribution&& make binaries&& mkdir /mnt/bin; cp -v bin/registry /mnt/bin"

Does this work for you?

[oglematt]

@wilminator That does! Thanks for tracking it down.

[Jean-Daniel]

18 months and still no fix for the stock image. Is this project dead ?

[gsaraf]

This is affecting me as well. What can I do to help move this forward?

[wilminator]

Everyone, I have an answer. TL,DR: If we want Docker to fix their registry image, this is the wrong place to post this. Go to registry incompatible with Let's Encrypt #96 and post there.

I did some more digging because this project has quite a few recent commits. Also, the one-liner above to launch a container to compile the registry binary shows that the project itself works fine- we get a working binary.

The problem most of us are facing is that the Docker registry image has a two year old version of the registry binary baked in, and this repo does not control that binary.

@gsaraf To fix this we need to use an issue with the main Docker GitHub repo and propose a merge into docker/distribution-library-image and hope they compile that down into a new image for us. The location this updated binary needs to get stashed is at https://github.com/docker/distribution-library-image/tree/master/amd64. However, this is the "wrong" way to fix this. The reality is that they should recompile their own source tree to fix this issue; it looks like we need to prod them to do it.

See registry incompatible with Let's Encrypt #96 and upvote like is suggested in the comments. We can also clone the repo, run their update, confirm their updated binary is ACME v2 compliant, and submit a merge request. However, I'm not holding my breath to see if their source is v2 compatible without any hacking.

[Jean-Daniel]

A better solution would be to stop relying on this link and docker official account and just published an updated image in a new official distribution account in docker hub.

While it is convenient to just have to use image registry:2, I don't mind switching to distribution/registry:2 if I can get an up-to-date image out of the box.

[Bec-k]

Still not fixed? 3 years have passed...

[milosgajdos]

Would you like to open a PR @denissabramovs ?

[Bec-k]

Nope, why would i? I just installed certbot myself, issued certs through it and configured crontab. Problem solved on my end.
It took me an hour, but it seems you can't fix it for 3 years...

[Jean-Daniel]

Would you like to open a PR @denissabramovs ?

I don't think it is possible to open a PR that performs a clean compilation of the project using a recent go toolchain and publish it…

This is all that is required to fix this issue.

[milosgajdos]

What are you missing in the toolchain that prevents you from doing this work @Jean-Daniel?

[Jean-Daniel]

What I am saying is that only the project maintainers can decide to release a new version and publish it (which is all that is needed to fix this issue, as go 1.18 is compatible with recent acme protocol). No PR can do that for you.

The last release was built with Go 1.16, which is why it fails to generate certificate.

On my side, that make a long time that I did it myself and published it in a public repository dedicated to that single task.

[milosgajdos]

@Jean-Daniel these are all fair comments. I've been trying to attend to various issues and PRs, but I agree we need more activity from maintainers. I'll be raising my voice more strongly at the next community call -- can't do more as is at the moment :-(

[olakmal]

Damn still no solution for this? 😥

[bhhaskin]

Still an issue

[milosgajdos]

@creekorful with the new release cut last night https://github.com/distribution/distribution/releases/tag/v3.0.0-alpha.1 would you mind taking it for a spin and test it.

[milosgajdos]

Closing for the lack of noise from the OP. Feel free to reopen.