kustomize localize command to pull all remote data

[monopole]

Some kustomize uses cases preclude network use, so it would be nice to capture the raw material from remote sources but still retain the option of performing different builds.

So perhaps

kustomize localize {target} {newDir}

would create a new directory (failing if it already existed), containing a localized, copied version of {target}, such that

kustomize build {target}
kustomize build {newDir}

would produce the same output, but the latter wouldn’t need the network to run.

The kustomization files under {newDir} would have their resources: fields rewritten to point to directories under {newDir}, named to not collide with any copies of {target} subdirectories. The contents of {newDir} would be the result of a graph traversal rooted at {target}.

Likewise, the helm directive would be rewritten to download and use a local chart.
See https://github.com/kubernetes-sigs/kustomize/blob/master/api/types/helmchartargs.go

This feature would benefit from first implementing #3979

[bgrant0607]

+1. Also could simplify access to private repos (requiring credentials) in some scenarios.

[marshall007]

@monopole I think this would be a fantastic addition to kustomize. We currently use kpt pkg commands to achieve roughly the same behavior. There are some friction points with our current approach because kpt is totally unopinionated in terms of the resources it manages, has no concept of package structure (as defined by a kustomization.yaml), and defines its own "package entrypoint" (i.e. a Kptfile vs Krmfile).

All that to say, a kustomize-aware alternative to kpt for package management would be tremendously useful to us. In addition to the use cases you outlined above, I think another very compelling reason to manage packages this way is that it makes it possible to consume/apply setters defined in remote packages.

I think a key piece of functionality necessary to make this feature sucessful would be the ability to intelligently merge upstream changes with changes made in the local package (when the remote is a git repo, that is). This is something that kpt pkg update does quite well and is super valuable. We use this extensively.

[bgrant0607]

If setters were changed to an on-the-fly transformer as discussed in #3953, then localize wouldn't be necessary in order to apply setter values to remote resources and merging on update wouldn't be necessary. I suggest that localize should just fetch, not merge, consistent with the existing fetch behavior during build.

[marshall007]

@bgrant0607 I definitely agree that applying setters on-the-fly is necessary as there should be no distinction between how setters (or any other functionality) behaves when referencing local vs remote resources.

That being said, I did not mean to conflate this with our desire for kustomize-aware package management utilities in general. We have found that it is simply easier to build and maintain kustomizations on top of resources that are declared locally. Also, because remote references are inherently mutable in most cases, we find explicitly requiring a human in the loop when pulling upstream changes to be a desirable property.

Ideally, something like kustomize localize as described (no 3-way merging) would be enough and an engineer would have no reason to modify the local copy of the remote package. In practice, however, it can be significantly easier to maintain minor manual edits to the remote resources that facilitate writing kustomizations/patches. This can include things like:

• adding a kustomization.yaml and populating resources: if one was not already present in the remote package
• adding setters that were not originally defined in the remote package
• updating hard-coded command flags to have their values sourced from env variables (more easily targeted by replacements)
• replacing static ConfigMap resources in favor of maintaining them exclusively as configMapGenerators

I see all of these use cases I'm describing as an extension of your statement in #3953:

I think of setters as an ad hoc transformer for cases where a dedicated transformer (like images, replicas) hasn't been built yet.

I don't want to go off on too much of a tangent here, but I would like to summarize by adding that we are not super pleased with the direction kpt has started to take as of v1, and hope to see standardization of a "universal package entrypoint" for tooling based on cli-utils. See:

• needless incompatibility between setter definitions in Improve setters API: SetterDefinition and SetterValues kptdev/kpt#985 (comment)
• recommended approach for using kpt live apply in conjunction with kustomize build now involves including the Kptfile in resources: due to lack of standardization of the ResourceGroup CRD Feature Request: Move ResourceGroup CRD from kpt to cli-utils cli-utils#364

This is our primary motivation for wanting to see the kpt pkg subcommands maintained in cli-utils (and exposed in kustomize).

[bgrant0607]

We experimented with integrating cli-utils actuation commands into kustomize, but the command architecture didn't make sense from an experience point of view. Integrating cli-utils into kubectl as originally planned (kubernetes/enhancements#810) encountered other challenges.

In the meantime, tools like Helm, Terraform, Pulumi, and dozens of others have already implemented their own mechanisms, as discussed in kubernetes-sigs/cli-utils#364 (comment), so the ship has kind of sailed.

As for "package entrypoint", packaging is out of scope for Kustomize and for Kubernetes:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/scope.md#examples-of-projects-and-areas-not-in-scope

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[natasha41575]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[natasha41575]

/remove-lifecycle stale
/lifecycle frozen

[annasong20]

/assign

[natasha41575]

@annasong20 and I are working internally on a proposal for this issue. I'm hopeful that we will have it ready in the next couple of weeks.

[bgrant0607]

Another reason localize may be useful: Remote bases may not work in some embedded uses of kustomize due to the git requirement:
https://github.com/kubernetes-sigs/kustomize/blob/master/api/internal/git/gitrunner.go#L25

[natasha41575]

Linking it here so we don't forget: for alpha, we should define what it means to "not handle" the helm fields: #4933 (comment)

[natasha41575]

Alpha version is done and will be available in the next release. Closing this now.