Trying to work with spegel for faster image pull times but fails since we have private tls certificates.

[badaldavda8]

Discussed in #10052

^{Originally posted by badaldavda8 April 9, 2024}
Describe the bug
We are using jfrog artifactory registry as our registry. we are using tls certificate authentication for resolving and accessing our registry endpoints. We were facing 500 error with mirror resolve retries exhausted for key while running a pod, it gave this error

│   Type     Reason     Age                   From               Message                                                                                                                                                                                                 │
│   ----     ------     ----                  ----               -------                                                                                                                                                                                                 │
│   Normal   Scheduled  29m                   default-scheduler  Successfully assigned large-pod-1 to ip-10-190-18-201.eu-west-1.compute.internal                                                                                                                    │
│   Normal   Pulling    27m (x4 over 29m)     kubelet            Pulling image "xxx"                                                                                                  │
│   Warning  Failed     27m (x4 over 29m)     kubelet            Failed to pull image "xxx": rpc error: code = Unknown desc = failed to pull and unpack image "xxx": failed to resolve reference "xxx": unexpected status from HEAD request to http://10.190.18.201:30020/v2/xxx 500 Internal Server Error                                                                                                                                                 │
│   Warning  Failed     27m (x4 over 29m)     kubelet            Error: ErrImagePull                                                                                                                                                                                     │
│   Warning  Failed     27m (x6 over 29m)     kubelet            Error: ImagePullBackOff                                                                                                                                                                                 │
│   Normal   BackOff    4m6s (x109 over 29m)  kubelet            Back-off pulling image xxx"                                                                                         │
│                                                                                                                                                                  

We have tls certificates. And we are trying to change the way we use certificates now. Instead of putting the certificate in /etc/containerd/certs.d/xyz.com/client.cert and /etc/containerd/certs.d/xyz.com/client.key we are trying to put the certificates in /etc/certs/xyz.com/client.cert and /etc/certs/xyz.com/client.key. The problem now is that when we go ahead and create hosts.toml -

server = 'https://registry-1.docker.io'

[host]

[host.'http://example.com:30020']
capabilities = ['pull', 'resolve']
client = ['/etc/certs/xxx/client.cert', '/etc/certs/xxx/client.key']

[host.'http://example.com:30021']
capabilities = ['pull', 'resolve']
client = ['/etc/certs/xxx/client.cert', '/etc/certs/xxx/client.key']

, it doesn't work the way it worked when we had it in /etc/containerd/certs.d/xyz.com/client.cert and /etc/containerd/certs.d/xyz.com/client.key.

The error is either -
failed to load X509 key pair: tls: found a certificate rather than a key in the PEM for the private key"

OR

Failed to request xyz.com: connect: no route to host

[badaldavda8]

With cri its straightforward --

--tlscert value path to TLS client certificate
--tlskey value path to TLS client key

https://github.com/containerd/containerd/blob/main/docs/hosts.md#ctr

But with hosts.toml this is not clear.

When I use cri with --tlscert and --tlskey, it works fine.

But with client = ["client.key", "client.cert"], it fails.

Any pointers/documentation to get an equivalent for hosts.toml would help here.

--tlscert value path to TLS client certificate
--tlskey value path to TLS client key