Can I have a load balanced cluster of registries, each of which uses its own local storage drive?

[chaosssliu]

Hi, my team is building our own container registry service. We have a list of intake container registries behind a load balancer. Due to the requirements, we are not able to have a non-dependency shared storage for the cluster of registries. In this case, can I have a load balanced cluster of registries and each of them uses its own local storage drive? Will this design crash the push/pull operations?

Another question is how can I complete my image scanning. Please share any resource that you know. Thank you for the help in advance.

[deleteriousEffect]

@chaosssliu the load balancer you described would have to route requests to the registry instance which should contain the data. For an off the shelf load balancer, this wouldn't work. Pushes and pulls are both composed of multiple HTTP requests, so without some way to send all those requests to a specific storage instance, you would not be able to reliably push or pull.

As for image scanning, I don't have a recommendation.

[chaosssliu]

Thank you @deleteriousEffect I did testing by running tons of pulls and pushes and observed the multiple HTTP requests of one action could be sent to different registry instances. In this case, if every time after I push an image to the cluster of registries, I merge all the repositories/blob data together into one folder, will this folder be able to serve as a complete registry storage and allow me to pull images from it without issues?

[deleteriousEffect]

In this case, if every time after I push an image to the cluster of registries, I merge all the repositories/blob data together into one folder, will this folder be able to serve as a complete registry storage and allow me to pull images from it without issues?

@chaosssliu you'd need constant replication between all storage backends — not just after image pushes. Near the end of a push, the registry makes checks to ensure that each blob or manifest referenced by the pushed manifest is present. If these went to different storage backends, the push would ultimately fail.

You'll also need to probably handle consistency during delete operations, as well as when an existing tag is used for a new image.

Honestly, I can't recommend attempting either approach discussed here. I would try to maintain a 1-to-1 relationship between each registry hostname and storage instance. You can accomplish this with a single storage backend, or you can have multiple registry hostnames.

[chaosssliu]

Got it. Thank you @deleteriousEffect