Clarify usage of registries.conf with different mirrors

[telmich]

I hope this is the right cri-o repo to report to, if not, please advise.

General Situation

• k8s clusters usually pull images from multiple registries like quay.io and dockerhub
• To be able to locally cache/proxy images, one needs to create a specific proxy per registry - resulting in https://proxy-for-docker-hub and https://proxy-for-quay-io
• You might have multiple proxies for each registry for redundancy like https://proxy-for-docker-hub-1 and https://proxy-for-docker-hub-2
• In the registries.conf the mirror list applies for all registries, which leads to a lot of false hits

Problem

We cannot specify that a certain mirror is only for certain paths. Thus resulting in a lot of incorrect accesses over the proxies.

Pratical example

The following configuration is taken from a sample kubernetes cluster:

[09:29] server27.place10:~# cat /etc/containers/registries.conf
unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "registry.hub.docker.com"

[[registry]]
prefix = "quay.io"
location = "quay.io"

# List of our mirrors
[[registry.mirror]]
location = "harbor.ungleich.svc.p10.k8s.ooo/quayio"

[[registry.mirror]]
location = "harbor.ungleich.svc.p10.k8s.ooo/dockerhub"

[[registry.mirror]]
location = "harbor.ungleich.svc.c2.k8s.ooo/dockerhub"

[[registry.mirror]]
location = "harbor.ungleich.svc.c2.k8s.ooo/quayio"

As you can see there are 2 quayio mirrors and 2 dockerhub mirrors. We plan to add 3 mirrors per official source for redundancy reasons. Thus per pull request cri-o might fall through 3 incorrect mirrors and then finally access the correct one.

Improvement

We suggest to add a new field, prefix to the mirror list:

[[registry.mirror]]
location = "harbor.ungleich.svc.c2.k8s.ooo/quayio"
prefix="quay.io"

This way only the mirrors with the right prefix would be contacted.

[vrothberg]

Thanks for reaching out, @telmich! You've found the right place for this kind of question.

If you have a dedicated mirror for Docker Hub and a dedicated mirror Quay, an exemplary config may look as follows:

[[registry]]
location = "docker.io"
[[registry.mirror]]
location = "docker.mirror.com"

[[registry]]
location = "quay.io"
[[registry.mirror]]
location = "quay.mirror.com"

Notice that the mirrors are not global for all registries but are associated with the previous registry. This way, when pulling from quay.io only quay.mirror.com will be consulted.

Does that answer your question?

[telmich]

THAT certainly answers the question! OMG, had we known this earlier... So in this regard I suggest to update the documentation to include the notion of "order matters" and maybe even including your example.

Thanks a lot for the quick clarification, much appreciated!

[vrothberg]

I opened #1525 to clarify the issue in the man pages.