Feature Request: Mutating Webhook Policy for Rewriting Image Paths

[Martin-Weiss]

Is your feature request related to a problem?

When ever we use an on-premise registry we have to modify the helm values for all helm charts to fetch the images from the on-premise registry instead of from the original - probably not reachable - external / internet located registy.

Modifying all the helm charts is a lot of effort so we are looking for some sort of automation.

In RKE2 or K3S we can use containerD registry mirror / registry rewrite feature - but that can not be used if multiple developers use shared clusters or shared registries or in case of public cloud Kubernetes clusters.

So we need some rewrite on a per cluster / per namespace / per project with some source filtering and target replacement similar to containerD rewrite.

Solution you'd like

A policy that can automatically replace image paths in deployments, daemon sets, jobs etc. basically everything that has images configured based on a set of rules. The rules might be namespace / project / cluster related so I guess we need a source filter and a target ruleset for replacements.

See also https://docs.rke2.io/install/containerd_registry_configuration#rewrites or https://github.com/Martin-Weiss/rke-cluster/blob/master/registries.yaml but with more ways to influence what source should be mutated to which target.

One more thing: for on-premise registries we also need authentication and authorization - so mabye the policy could also add the required registry secret and configuration for using it as well.

Alternatives you've considered

containerD rewrite but that does not give per-namespace or per-deployment rewrites.

See also https://github.com/phenixblue/imageswap-webhook which might be an alternative but adding another thing if we have kubewarden, already... might be a challenge.

Anything else?

One of the challenges of such a mutating webhook might be the combination with CI/CD tools like Fleet or ArgoCD where a constant compare between declared state and real state is done (reconciliation loop). If a mutating webhook changes what gets deployed / defined with i.e. Fleet - fleet might get into an endless loop trying to fix what the webhook changed. Not sure how this could be addressed with mutating webhooks in general.