Authors:
- annasong20
Reviewers:
- natasha41575
- KnVerey
Status: implementable
The kustomize localize
command creates a “localized” copy, of both the target kustomization and files target
references, in which the kustomization files contain, instead of remote references, local paths to their downloaded
locations. The command is part of an effort to enable kustomize build
to run without network access.
monopole originally proposed the command kustomize localize
in this issue.
Users run kustomize build
in many environments with limited network access. For example, CI/CD pipelines often only
have access to the internal network. Server-side applications like Config Sync are concerned with the security
vulnerabilities of git, which kustomize build
uses to fetch remote files.
These use cases would benefit from a kustomize solution that downloads all remote files that a kustomize build
target
references, into a copy of target that references the downloaded files instead. Admins could upload the localized copy
to an internal repo so that pipelines and applications can run kustomize build
on the copy without a network
dependency.
The proposed command nearly achieves the solution by downloading all remote files directly referenced by the target or
by a recursively referenced kustomization file. The command also downloads remote exec binaries of referenced KRM
functions, which are the only potential source of remote files other than kustomizations. The only remote files that
this proposal does not cover and that kustomize build
still needs to run are remote images and custom fields in KRM
functions. Downloaded images would live only in local caches and thus, are not worth localizing. Kustomize cannot
currently identify custom fields, though this may change with one of the proposed solutions
in issue #4154.
The proposed command has the added benefit of increasing user confidence in the integrity of their kustomization builds. Locally downloaded files, unlike urls, give users full control of file content. At the same time, the command does this without modifying the original kustomization so that users can always run the command on the original again to fetch upstream changes.
Goals:
-
This command should localize
- all remote files that a kustomization file directly references
- remote exec binaries of referenced KRM functions
This command achieves this goal if, in the absence of remote images and custom fields in KRM functions,
kustomize build
can run on the localized copy without network access.
Non-goals:
- This command should not localize remote images or custom fields in KRM functions.
- This command should not copy files that the target kustomization does not reference.
- This command should not serve as a package manager.
The command takes the following form:
kustomize localize target newDir [-s scope] [-n]
where the arguments are:
target
: a directory with a top-level kustomization file that kustomize will localize; can be a path to a local directory or a url to a remote directorynewDir
: optional destination directory of the localized copy oftarget
; if not specified, the destination is a directory namedlocalized-{target}
in the same directory asscope
and the flags are:
-s
,--scope
scope
: optional root directory, files outside which kustomize is not allowed to copy and localize; if not specified, takes on value oftarget
-n
,--no-verify
: do not verify that the outputs ofkustomize build
fortarget
andnewDir
are the same after localization
The command creates a copy of the target
kustomization and the local files that target
references at newDir
. We
define the "files that target
references" as:
- kustomization files that
target
directly or transitively references - configuration files that referenced kustomization files reference
- exec binaries of referenced KRM functions
Here, configuration file means a non-kustomization yaml file. The command cannot run on target
s that need
the --load-restrictor LoadRestrictionsNone
flag on kustomize build
. The command only copies referenced files that
reside inside scope
.
The command localizes the copy of target
at newDir
by downloading all remote files that target
references. Users do not have executable permission for downloaded exec binaries that KRM functions reference and for
each such exec binary, the command will print a warning message to that effect.
The command creates a new localized-files
directory, next to the file that referenced the downloaded files, to hold
said files. Inside localized-files
, the downloads are located on path:
domain / organization / repo / version / path/to/file/in/repo
where version
corresponds to the ref
query string parameter in the url, though ideally version
ia a stable tag as
opposed to a branch.
The command replaces remote references in newDir
with the local paths of the downloaded files. To help ensure
that newDir
is a clean copy, the command additionally overwrites absolute path references into target
to point
to newDir
.
As a convenience to the user, in the absence of the --no-verify
flag, the command automatically tries to
run kustomize build
, without any flags, on target
and the localized newDir
to compare their outputs. The command
indicates success if the outputs match and throws an error with the diff summary otherwise. This check, however, is not
useful for certain target
s, including those that need flags to build. In these cases, the command prints next steps
that users can follow to check the output themselves. For example, for target
s that reference KRM functions with a
remote exec binary, the command suggests the user:
- add executable permissions for the downloaded exec binaries in
newDir
that the user trusts - run
kustomize build
with flags--enable-alpha-plugins --enable-exec
and self-verify the outputs
Error cases:
target
does not have a top-level kustomization filekustomize build
needs--load-restrictor LoadRestrictionsNone
to run ontarget
newDir
already existsscope
does not containtarget
target
references a local path that traverses outside ofscope
- remote reference does not have a
version
localized-files
directory already exists- kustomization file is malformed
- cycle of kustomization file references exists
kustomize build
produces different output fortarget
andnewDir
in the absence of--no-verify
Depending on feedback, we may add an --overwrite
flag in the future to allow users to update an existing newDir
by
running the command again.
Warning cases:
- KRM function references remote exec binary, in which case the downloaded exec binary is not executable
- KRM function has container image that the user might not have locally
My company’s CI/CD pipeline currently pulls an example
directory from our internal package management site. I want the
CI/CD pipeline to additionally run kustomize build example/overlay
. My setup looks like this:
└── example
├── overlay
│ └── kustomization.yaml
└── base
└── kustomization.yaml
# example/overlay/kustomization.yaml
resources:
- ../base
- https://raw.githubusercontent.com/kubernetes-sigs/kustomize/v1.0.6/examples/helloWorld/deployment.yaml
# example/base/kustomization.yaml
resources:
- github.com/kubernetes-sigs/kustomize/examples/multibases?ref=v1.0.6
- https://raw.githubusercontent.com/kubernetes-sigs/kustomize/v1.0.6/examples/helloWorld/configMap.yaml
I get an error from kustomize build
in the pipeline because my configurations have remote references, but my CI/CD
pipeline does not have external network access.
Fortunately, I remember that I can run kustomize localize
on example/overlay
on my local machine. I can then upload
the localized directory to my company’s internal package management site for the CI/CD pipeline to pull and build
instead. I run
kustomize localize example/overlay -s example
, where my target
is example/overlay
, I accept the default location
and name of newDir
, and I expand my scope
to example
because example/overlay
references example/base
. I get
the following output:
$ kustomize localize example/overlay -s example
SUCCESS: example/overlay, localized-overlay produce same kustomize build output
├── example # old kustomization directory
│ ├── overlay
│ │ └── kustomization.yaml
│ └── base
│ └── kustomization.yaml
└── localized-overlay # the new, localized kustomization directory
├── kustomization.yaml
├── base
│ ├── kustomization.yaml
│ └── localized-files
│ └── github.com
│ └── kubernetes-sigs
│ └── kustomize
│ └── v1.0.6
│ └── examples
│ ├── helloWorld
│ │ └── configMap.yaml
│ └── multibases
│ ├── base
│ │ └── pod.yaml
│ ├── dev
│ ├── kustomization.yaml
│ ├── production
│ └── staging
└── overlay
├── kustomization.yaml
└── localized-files
└── github.com
└── kubernetes-sigs
└── kustomize
└── v1.0.6
└── examples
└── helloWorld
└── deployment.yaml
# localized-overlay/overlay/kustomization.yaml
resources:
- ../base
- ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/deployment.yaml
# localized-overlay/base/kustomization.yaml
resources:
- ./localized-files/github.com/kubernetes-sigs/kustomize/examples/multibases
- ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/configMap.yaml
Now, I upload localized-overlay
from my local setup to my company’s internal package management site. I change the
commands in my CI/CD pipeline to pull localized-overlay
before running kustomize build localized-overlay
, and the
command executes successfully!
One could argue that while uploading the localized newDir
to a repository, a user could accidentally leak Secrets that
were originally remote in a more private repo. This event is not very likely, as the servers that the user intends to
consume the localized kustomizations often only have access to internal, private networks and repos. Nonetheless, users
should have enough context in the case of Secrets to make the right decision. Users should have a basic understanding of
the files that their target
kustomizations reference and of the files that they plan to upload to repos. Secret
configurations are also not too difficult to identify.
Exec binaries that KRM functions reference are a different story. kustomize localize
downloads remote exec binaries
that, if malicious, are capable of almost anything during subsequent kustomize build
calls. The command mitigates this
risk by leaving these downloaded exec binaries without executable permissions and warning the user, as mentioned in
Proposal. kustomize build
can only run the exec binary after the user deems the binary safe and changes its
permissions.
Still another risk may be that if a user's kustomization tree is large, kustomize localize
may be copying files from
unexpected locations. The command mitigates this risk with the scope
flag. If not set, kustomize localize
only
copies files in target
. Otherwise, the user specifies scope
and understands that kustomize localize
only copies
files in scope
. The qualification that the command can only localize target
s that follow load restrictions helps
mitigate this risk as well.
N/A
Large kustomization trees slows the performance of kustomize localize
. These trees can have large local subtrees, have
large remote subtrees, be deeply nested, or be wide, with each overlay referencing multiple bases. Regardless of the
cause, large kustomization trees inevitably takes longer to copy and download. Parts of the kustomize code are not
thread-safe, which precludes parallel execution.
On a separate note, a chain of remote kustomization directories in which the current kustomization file references the
next remote kustomization directories could create a newDir
with deeply nested localized-files
. This directory
structure would impede users’ navigation of newDir
. However, this scenario should be unlikely as most kustomizations
only consist of a few layers.
The creation of the localized-files
directory local to the referencing kustomization file additionally prevents the
different layers of kustomization files from sharing the same copy of the remote files. Following the same logic,
different potential target
directories cannot share copies either.
Users whose layered kustomizations form a complex directory tree structure may have a hard time finding an
appropriate scope
. However, many kustomizations exist in repositories, allowing the user to easily choose the repo
root as a valid scope
. The warning messages that kustomize localize
outputs for reference paths that extend
beyond scope
should also help.
-
Instead of downloading into
newDir
,kustomize localize
could download all remote files into a directory specified by some global environment variable (like in Golang), which would preclude deeply nested directories and allow different kustomization files to share configurations. On top of that, ifkustomize build
had the added functionality to check for previous downloads of remote references at said global location,kustomize localize
would not need to overwrite the remote references intarget
to the local downloads. As a result,kustomize localize
would need to neither write totarget
nor copytarget
intonewDir
. The user would not need to specify ascope
either.Despite its advantages, the alternative design violates the self-contained nature of each kustomize layer. Users would be unable to upload a fully localized kustomization directory in version control. Furthermore, this alternative complicates the existing kustomize workflow by requiring the setup of global environment variables.
-
The command could, instead of making a copy, modify
target
directly. However, users would not have an easy way to undo the command, which is undesirable. -
Instead of requiring the user to specify a second argument
scope
, the command could by definition limit its copying totarget
. However, in the case of Story 1, the command would force the user to settarget
toexample
in order to includeexample/base
in the localization ofexample/overlay
. The user would then have to create a kustomization file atexample
that points toexample/overlay
under theresources
field. The creation of the kustomization file solely for this purpose is messy and more work for the user.
This command will have at least alpha and GA releases. Depending on user feedback, we may add a beta.
This release will not support
- load restrictions check
- KRM functions
- absolute paths
target
kustomization with reference cycles- any verification in the form of
--no-verification
flag or automatically runningkustomize build
at the end of localization
The entire command will be new in the alpha release, and so will not require an alpha flag. The command will not be
available in kubectl kustomize
either as kubectl only has kustomize build
builtin.
This release should have all features documented in this proposal. Though, we may make changes based on user feedback.