Implications of indexing a layer store by DiffID instead of ChainID?

[ayazhafiz]

Hi folks,

I am trying to understand the implications of having a container management tool index an image layer store by layer DiffIDs instead of ChainIDs.

As far as I understand, all major container management tools (I've looked at Docker, Podman, containerd, and nerdctl - the last of which looks to use containerd's implementation) index image layers in the layer store by ChainID. I understand that the ChainID is useful for uniquely identifying the base filesystem of an image; however, is there an inherent reason it is not sufficient (for security reasons or otherwise) to index a layer store itself by layers' DiffID?

I ask specifically because I have a use case that requires storing many images with a number of shared layers (as determined by their diff ID), but with those layers in different orders (and hence different chain IDs). I am considering a layer store implementation that would index by diff ID to enable structural sharing, but it's unclear to me if doing so has significant consequences, or if there is a better way.

Apologies if this is the wrong forum to ask this question, or if it has been answered elsewhere. I have been unable to find any online discussion of this question specifically, but if there is a well-known answer or a better forum, I would appreciate your direction.

[neersighted]

cc @stevvooe; I can try to explain it, but I think a more first-party source might have a better layman's explanation.