Docker Distribution (Private Container Registry) is not working with S3 compatible storage

[sky29]

Hi,

I am trying to run Docker Distribution (Private Container Registry) with S3 compatible storage. I am not using AWS S3, but I tried with Google Cloud Storage Interoperability layer & MinIO: https://console.cloud.google.com/storage/settings;tab=interoperability

Below are the steps I executed:

1. create a htpasswd to store registry secrets

docker run \
  --entrypoint htpasswd \
  httpd:2 -Bbn testuser testpassword > auth/htpasswd

2. config.yml

version: 0.1
storage:
  s3:
    accesskey: <GCS-ACCESS-KEY>
    secretkey: <GCS-SECRET-KEY>
    region: <GCS-BUCKET-REGION>
    regionendpoint: https://storage.googleapis.com
    bucket: <GCS-BUCKET>
    chunksize: 5242882
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd

3. created let's encrypt certificate (using certbot) and derive domain.crt, domain.key

cd <LOCAL-PATH>/certs
cp privkey.pem domain.key
cat cert.pem chain.pem > domain.crt
chmod 777 domain.crt
chmod 777 domain.key

4. docker run

docker run -d -p 443:5000 --name s3-registry \
-v <LOCAL-PATH>:/auth \
-v <LOCAL-PATH>/config.yml:/etc/docker/registry/config.yml \
-v <LOCAL-PATH>/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
registry:2.8.1

6. verify

Container Process: docker ps --> this is running fine
UI/Browser: https://localhost/v2/_catalog on browser --> this is also running fine, it ask for basic auth and succeed
CLI: docker login -u testuser -p testpassword localhost:443 --> this is also running fine, it ask for basic auth and succeed

7. So everything looks good, but when i try to push image, it gives following error:

Using default tag: latest
The push refers to repository [localhost:443/my-ubuntu]
1251204ef8fc: Retrying in 1 second 
47ef83afae74: Retrying in 1 second 
df54c846128d: Retrying in 1 second 
be96a3f634de: Retrying in 1 second 
received unexpected HTTP status: 500 Internal Server Error

and also when i check container logs, i see below error:

172.17.0.1 - - [20/Oct/2022:05:40:44 +0000] "POST /v2/my-ubuntu/ubuntu/blobs/uploads/ HTTP/1.1" 500 123 "" "docker/20.10.6 go/go1.13.15 git-commit/8728dd2 kernel/5.10.25-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.6 \\(darwin\\))"
172.17.0.1 - - [20/Oct/2022:05:40:44 +0000] "POST /v2/my-ubuntu/ubuntu/blobs/uploads/ HTTP/1.1" 500 123 "" "docker/20.10.6 go/go1.13.15 git-commit/8728dd2 kernel/5.10.25-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.6 \\(darwin\\))"
time="2022-10-20T05:40:44.828571783Z" level=error msg="response completed with error" auth.user.name=testuser err.code=unknown err.detail="s3aws: InvalidArgument: Invalid argument.
	status code: 400, request id: , host id: " err.message="unknown error" go.version=go1.16.15 http.request.host="localhost:443" http.request.id=7e5d5ba7-95a3-4807-930a-3c632bf35e33 http.request.method=POST http.request.remoteaddr="172.17.0.1:57006" http.request.uri="/v2/my-ubuntu/ubuntu/blobs/uploads/" http.request.useragent="docker/20.10.6 go/go1.13.15 git-commit/8728dd2 kernel/5.10.25-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.6 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=385.930525ms http.response.status=500 http.response.written=123 vars.name="my-ubuntu/ubuntu" 

can anyone help? Same S3 credentials (for GCS Interoperability layer) are working fine with other solutions (like chartmuseum etc.).
Is Docker S3 Storage [ mentioned here: https://docs.docker.com/registry/storage-drivers/s3/ ] works with any S3 compatible storage or only AWS S3 Storage ?

I also tried setting below environment variables, but getting same error:

REGISTRY_STORAGE
REGISTRY_STORAGE_S3_ACCESSKEY
REGISTRY_STORAGE_S3_SECRETKEY
REGISTRY_STORAGE_S3_REGION
REGISTRY_STORAGE_S3_REGIONENDPOINT
REGISTRY_STORAGE_S3_BUCKET
REGISTRY_AUTH
REGISTRY_AUTH_HTPASSWD_REALM
REGISTRY_AUTH_HTPASSWD_PATH

[milosgajdos]

Can you turn on debug log, please. Also, can you please use the edge build?

[sky29]

@milosgajdos

I enabled the debug and found below error in logs:

time="2022-10-24T22:16:15.546310106Z" level=error msg="response completed with error" auth.user.name=<REPLACE-ME> err.code="blob unknown" err.detail=sha256:1aa843a1090f94c0e1419850cc9a0b3798f50c380562dc03e7d21f155ab38906 err.message="blob unknown to registry" go.version=go1.16.15 http.request.host=<REPLACE-ME> http.request.id=e031cf90-5665-44a8-998b-94b08e55a0a7 http.request.method=HEAD http.request.remoteaddr="<REPLACE-ME>:24404" http.request.uri="/v2/<REPLACE-ME>/blobs/sha256:1aa843a1090f94c0e1419850cc9a0b3798f50c380562dc03e7d21f155ab38906" http.request.useragent="docker/20.10.6 go/go1.13.15 git-commit/8728dd2 kernel/5.10.25-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.6 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=179.643683ms http.response.status=404 http.response.written=157 vars.digest="sha256:1aa843a1090f94c0e1419850cc9a0b3798f50c380562dc03e7d21f155ab38906" vars.name=<REPLACE-ME>

FYI: in above log - i removed some sensitive information and replaced it with REPLACE-ME manually

I can see one similar GitHub Issue:
#1636
and solution in comment: #1636 (comment)

I am running container registry without HTTPS (insecure registry) because I am planning to add NGINX proxy in front of it with HTTPS termination - so not sure above solution is applicable in my case (though I will try it).