From c74f06125172ec95aa62254ec1ce0e033c3cc114 Mon Sep 17 00:00:00 2001 From: Anna Song Date: Tue, 19 Apr 2022 00:53:52 -0400 Subject: [PATCH 1/4] Propose kustomize localize Write mini KEP proposal for new command kustomize localize. --- proposals/22-04-localize-command.md | 296 ++++++++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 proposals/22-04-localize-command.md diff --git a/proposals/22-04-localize-command.md b/proposals/22-04-localize-command.md new file mode 100644 index 0000000000..604ec00727 --- /dev/null +++ b/proposals/22-04-localize-command.md @@ -0,0 +1,296 @@ +# Localize Command + +**Authors**: +- annasong20 + +**Reviewers**: +- natasha41575 +- KnVerey + +**Status**: implementable + +## Summary + +The `kustomize localize` command creates a “localized” copy of the target kustomization directory in which the layered +kustomization files contain local, instead of remote, references to downloaded files. The command is part of an effort +to enable `kustomize build` to run without network access. + +## Motivation + +monopole originally proposed the command `kustomize localize` +in [this issue](https://github.com/kubernetes-sigs/kustomize/issues/3980). + +Users run `kustomize build` in many environments with limited network access. For example, CI/CD pipelines often only +have access to the internal network. Server-side applications like Config Sync are concerned with the security +vulnerabilities of git, which `kustomize build` uses to fetch remote files. + +These use cases would benefit from a kustomize solution that downloads all remote files that a `kustomize build` target +references, into a copy of the target that references the downloaded files instead. `kustomize build` would then be able +to run on the copy without a network dependency. + +This proposal nearly achieves the solution by downloading all remote files directly referenced by the target or by a +recursively referenced kustomization file. The only remote files not covered by this proposal and still needed for +`kustomize build` to run are those in KRM functions. However, KRM functions are third party to kustomize, and so will be +out of scope for `kustomize localize`. + +**Goals:** +1. This command should localize all remote files that a kustomization file directly references. This command will have + achieved this goal if, in the absence of remote input files to KRM functions, `kustomize build` can run on the + localized copy without network access. + +**Non-goals:** +1. This command should not localize remote input files to KRM functions. +1. This command should not serve as a package manager. + +## Proposal + +The command takes the following form: + +
+kustomize localize target newDir
+
+ +where the arguments are: + +* `target`: a directory with a top-level kustomization file that kustomize will localize; can be a path to a local + directory or a url to a remote directory +* `newDir`: destination of the localized copy of `target` + * if `target` is local, `newDir` must be a directory name, as it will be located in the same directory as `target` + to preserve relative path references + * if `target` is remote, `newDir` must be a directory path + +The command creates a copy of `target` at `newDir` in which each kustomization file, from the top-level to any +recursively referenced, has local paths to downloaded files instead of remote references for the following kustomization +fields: + +* `resources` +* `components` +* `bases` +* `openapi:` +     `paths` + +A new `localized-files` directory holds the downloaded files (or directory) +at: + +
+remote-host / organization / repo / version / path-to-file-in-repo
+
+ +Each `localized-files` directory is located in the same directory as the kustomization file that referenced the +downloaded files. + +To help ensure that `newDir` is a clean copy, the command overwrites every absolute path into `target` to point +to `newDir` before processing the path. + +**Error cases**: +* `target` does not have a top-level kustomization file +* `newDir` already exists +* `localized-files` directory already exists +* remote reference does not have a version +* kustomization file is malformed +* cycle of kustomization file references exists + +**Warning cases**: +* `newDir` refers to `base` kustomization layer outside of `newDir`, and `base` has remote references +* KRM function has container image or exec binary that the user might not have locally + +If the command runs without any errors, `kustomize build` on `target` and `newDir` should produce the same output. + +### User Stories + +#### Story 1 + +My company’s CI/CD pipeline currently pulls an `example` directory from our internal package management site. I want the +CI/CD pipeline to additionally run `kustomize build example/overlay`. My setup looks like this: +``` +└── example + ├── overlay + │ └── kustomization.yaml + └── base + └── kustomization.yaml +``` +``` +# example/overlay/kustomization.yaml +resources: + - ../base + - https://raw.githubusercontent.com/kubernetes-sigs/kustomize/v1.0.6/examples/helloWorld/deployment.yaml +``` +``` +# example/base/kustomization.yaml +resources: + - github.com/kubernetes-sigs/kustomize/examples/multibases?ref=v1.0.6 + - https://raw.githubusercontent.com/kubernetes-sigs/kustomize/v1.0.6/examples/helloWorld/configMap.yaml +``` + +I get an error from `kustomize build` in the pipeline because my configurations have remote references, but my CI/CD +pipeline does not have external network access. + +Fortunately, I remember that I can run `kustomize localize` on `example/overlay` on my local machine. I can then upload +the localized directory to my company’s internal package management site for the CI/CD pipeline to pull and build +instead. I run +`kustomize localize example/overlay example/localized-overlay`, but I get the following error: + +``` +$ kustomize localize example/overlay example/localized-overlay +Warning: kustomization directory example/overlay refers to remote resources in example/base +``` + +because I forgot that `kustomize localize` can only localize remote references originating from within the +`target`, `example/overlay`. Therefore, command could not localize the remote references in `example/base`, +which `example/overlay/kustomization.yaml` locally references. `kustomize build example/localized-overlay` will still +run correctly outside the CI/CD pipeline, but will still require network access: + +``` +└── example + ├── overlay + │ └── kustomization.yaml + ├── localized-overlay + │ ├── kustomization.yaml + │ └── localized-files + │ └── github.com + │ └── kubernetes-sigs + │ └── kustomize + │ └── v1.0.6 + │ └── examples + │ └── helloWorld + │ └── deployment.yaml + └── base + └── kustomization.yaml +``` + +#### Story 2 + +I am back again from **Story 1**, but this time ready to localize `base` too, instead of just the `overlay` directory. +To localize both, I change my `target` argument from `example/overlay` to `example` to make a copy of both directories. +I add a top-level kustomization file to `example` with the following commands: + +``` +kustomize init; kustomize edit add resource overlay +``` + +I have deleted `example/localized-overlay` from **Story 1**. My setup now looks like this: +``` +└── example + ├── kustomization.yaml + ├── overlay + │ └── kustomization.yaml + └── base + └── kustomization.yaml +``` +``` +# example/kustomization.yaml +resources: + - ./overlay +``` + +with all other files having the same contents. After I run `kustomize localize example localized-example`, I get the +following: +``` +├── example # the old kustomization directory +│ ├── kustomization.yaml +│ ├── overlay +│ │ └── kustomization.yaml +│ └── base +│ └── kustomization.yaml +└── localized-example # the new, localized kustomization directory + ├── kustomization.yaml + ├── base + │ ├── kustomization.yaml + │ └── localized-files + │ └── github.com + │ └── kubernetes-sigs + │ └── kustomize + │ └── v1.0.6 + │ └── examples + │ ├── helloWorld + │ │ └── configMap.yaml + │ └── multibases + │ ├── base + │ │ └── pod.yaml + │ ├── dev + │ ├── kustomization.yaml + │ ├── production + │ └── staging + └── overlay + ├── kustomization.yaml + └── localized-files + └── github.com + └── kubernetes-sigs + └── kustomize + └── v1.0.6 + └── examples + └── helloWorld + └── deployment.yaml +``` +``` +# localized-example/overlay/kustomization.yaml +resources: + - ../base + - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/deployment.yaml +``` +``` +# localized-example/base/kustomization.yaml +resources: + - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/multibases + - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/configMap.yaml +``` + +Now, I upload `localized-example` from my local setup to my company’s internal package management site. I change the +commands in my CI/CD pipeline to pull `localized-example` before running `kustomize build localized-example`, and the +command executes successfully! + +### Risks and Mitigations +N/A + +### Dependencies +N/A + +### Scalability + +A chain of remote kustomization directories in which the current kustomization file references the next remote +kustomization directories could create a `newDir` with deeply nested `localized-files`. This directory structure would +impede users’ navigation of `newDir`. However, this scenario should be unlikely as most kustomizations only consist of a +few layers. + +The creation of the `localized-files` directory local to the referencing kustomization file additionally prevents the +different layers of kustomization files from sharing the same copy of the remote files. Following the same logic, +different potential `target` directories cannot share copies either. + +## Drawbacks + +Users, like the one in the **User Stories** section, whose kustomization layers are in sibling directories need to +perform the extra step of creating a top-level kustomization file. However, many existing kustomize use cases also +require this step, and as shown in **Story 2**, users can create kustomization files relatively easily via command line. + +## Alternatives + +* Instead of downloading into `newDir`, `kustomize localize` could download all remote files into a directory specified + by some global environment variable (like in Golang), which would preclude deeply nested directories and allow + different kustomization files to share configurations. On top of that, if `kustomize build` had the added + functionality to check for previous downloads of remote references at said global location, `kustomize localize` would + not need to overwrite the remote references in `target` to the local downloads. As a result, `kustomize localize` + would need to neither write to `target` nor copy `target` into `newDir`. The user would not need to create a top-level + kustomization file either.

+ + Despite its advantages, the alternative design violates the self-contained nature of each kustomize layer. Users would + be unable to upload a fully localized kustomization directory in version control. Furthermore, this alternative + complicates the existing kustomize workflow by requiring the setup of global environment variables. + +* The command could, instead of making a copy, modify `target` directly. However, users would not have an easy way to + undo the command, which is undesirable. + +## Rollout Plan +This command will have at least alpha and GA releases. Depending on user feedback, we may add a beta. + +### Alpha + +This release will limit the depth of nested `localized-files` to 1 layer. In other words, the command will ignore remote +kustomization directory references that originate from within a `localized-files` directory. In addition, this release +will ignore KRM functions. The command will not output warnings for remote KRM images or exec binaries. + +The entire command will be new in the alpha release, and so will not require an alpha flag. The command will not be +available in `kubectl kustomize` either as kubectl only has `kustomize build` builtin. + +### Beta/GA + +This release should have all features documented in this proposal. Though, we may make changes based on user feedback. From 48fb056c728f0b481132a3e00883d614c704d5c8 Mon Sep 17 00:00:00 2001 From: Anna Song Date: Fri, 6 May 2022 07:50:37 -0700 Subject: [PATCH 2/4] Incomplete commit --- proposals/22-04-localize-command.md | 32 ++++++++++++++++++----------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/proposals/22-04-localize-command.md b/proposals/22-04-localize-command.md index 604ec00727..616dfc629f 100644 --- a/proposals/22-04-localize-command.md +++ b/proposals/22-04-localize-command.md @@ -1,9 +1,11 @@ # Localize Command -**Authors**: +**Authors**: + - annasong20 -**Reviewers**: +**Reviewers**: + - natasha41575 - KnVerey @@ -11,9 +13,9 @@ ## Summary -The `kustomize localize` command creates a “localized” copy of the target kustomization directory in which the layered -kustomization files contain local, instead of remote, references to downloaded files. The command is part of an effort -to enable `kustomize build` to run without network access. +The `kustomize localize` command creates a “localized” copy of the target kustomization and any files target transitively +references, in which the kustomization files contain local, instead of remote, references to downloaded files. The +command is part of an effort to enable `kustomize build` to run without network access. ## Motivation @@ -25,35 +27,41 @@ have access to the internal network. Server-side applications like Config Sync a vulnerabilities of git, which `kustomize build` uses to fetch remote files. These use cases would benefit from a kustomize solution that downloads all remote files that a `kustomize build` target -references, into a copy of the target that references the downloaded files instead. `kustomize build` would then be able -to run on the copy without a network dependency. +references, into a copy of target that references the downloaded files instead. The copy would include the target and +any files target transitively references.`kustomize build` would then be able to run on the copy without a network +dependency. This proposal nearly achieves the solution by downloading all remote files directly referenced by the target or by a -recursively referenced kustomization file. The only remote files not covered by this proposal and still needed for -`kustomize build` to run are those in KRM functions. However, KRM functions are third party to kustomize, and so will be -out of scope for `kustomize localize`. +transitively referenced kustomization file. The only remote files not covered by this proposal and still needed for +`kustomize build` to run are those in KRM functions. The command copies local exec binaries of KRM functions only as +part of copying target. The actual localization only applies to kustomization resources. KRM functions are third party +to kustomize, and thus KRM function remotes are out of scope for `kustomize localize`. **Goals:** + 1. This command should localize all remote files that a kustomization file directly references. This command will have achieved this goal if, in the absence of remote input files to KRM functions, `kustomize build` can run on the localized copy without network access. **Non-goals:** + 1. This command should not localize remote input files to KRM functions. -1. This command should not serve as a package manager. +2. This command should not copy files that the target kustomization does not reference. +3. This command should not serve as a package manager. ## Proposal The command takes the following form: 
-kustomize localize target newDir
+kustomize localize target scope newDir
where the arguments are: * `target`: a directory with a top-level kustomization file that kustomize will localize; can be a path to a local directory or a url to a remote directory +* `scope`: `target` or a directory that contains `target` * `newDir`: destination of the localized copy of `target` * if `target` is local, `newDir` must be a directory name, as it will be located in the same directory as `target` to preserve relative path references From 17cf626887c34f5de3c94fddf3178318c7241bc6 Mon Sep 17 00:00:00 2001 From: Anna Song Date: Mon, 9 May 2022 10:38:34 -0700 Subject: [PATCH 3/4] Update proposal based on Katrina's feedback Changes are summarized in comments in PR --- proposals/22-04-localize-command.md | 231 ++++++++++++++++------------ 1 file changed, 131 insertions(+), 100 deletions(-) diff --git a/proposals/22-04-localize-command.md b/proposals/22-04-localize-command.md index 616dfc629f..003aa8398f 100644 --- a/proposals/22-04-localize-command.md +++ b/proposals/22-04-localize-command.md @@ -13,9 +13,9 @@ ## Summary -The `kustomize localize` command creates a “localized” copy of the target kustomization and any files target transitively -references, in which the kustomization files contain local, instead of remote, references to downloaded files. The -command is part of an effort to enable `kustomize build` to run without network access. +The `kustomize localize` command creates a “localized” copy, of both the target kustomization and files target +references, in which the kustomization files contain, instead of remote references, local paths to their downloaded +locations. The command is part of an effort to enable `kustomize build` to run without network access. ## Motivation @@ -27,25 +27,29 @@ have access to the internal network. Server-side applications like Config Sync a vulnerabilities of git, which `kustomize build` uses to fetch remote files. These use cases would benefit from a kustomize solution that downloads all remote files that a `kustomize build` target -references, into a copy of target that references the downloaded files instead. The copy would include the target and -any files target transitively references.`kustomize build` would then be able to run on the copy without a network +references, into a copy of target that references the downloaded files instead. Admins could upload the localized copy +to an internal repo so that pipelines and applications can run `kustomize build` on the copy without a network dependency. -This proposal nearly achieves the solution by downloading all remote files directly referenced by the target or by a -transitively referenced kustomization file. The only remote files not covered by this proposal and still needed for -`kustomize build` to run are those in KRM functions. The command copies local exec binaries of KRM functions only as -part of copying target. The actual localization only applies to kustomization resources. KRM functions are third party -to kustomize, and thus KRM function remotes are out of scope for `kustomize localize`. +The proposed command nearly achieves the solution by downloading all remote files directly referenced by the target or +by a recursively referenced kustomization file. The command also downloads remote exec binaries of referenced KRM +functions, which are the only potential source of remote files other than kustomizations. The only remote files that +this proposal does not cover and that `kustomize build` still needs to run are remote images and custom fields in KRM +functions. Downloaded images would live only in local caches and kustomize cannot know the form that custom fields will +take. Thus, neither is worth localizing. **Goals:** -1. This command should localize all remote files that a kustomization file directly references. This command will have - achieved this goal if, in the absence of remote input files to KRM functions, `kustomize build` can run on the - localized copy without network access. +1. This command should localize + * all remote files that a kustomization file directly references + * remote exec binaries of referenced KRM functions + + This command achieves this goal if, in the absence of remote images and custom fields in KRM + functions, `kustomize build` can run on the localized copy without network access. **Non-goals:** -1. This command should not localize remote input files to KRM functions. +1. This command should not localize remote images or custom fields in KRM functions. 2. This command should not copy files that the target kustomization does not reference. 3. This command should not serve as a package manager. @@ -61,37 +65,35 @@ where the arguments are: * `target`: a directory with a top-level kustomization file that kustomize will localize; can be a path to a local directory or a url to a remote directory -* `scope`: `target` or a directory that contains `target` -* `newDir`: destination of the localized copy of `target` - * if `target` is local, `newDir` must be a directory name, as it will be located in the same directory as `target` - to preserve relative path references - * if `target` is remote, `newDir` must be a directory path +* `scope`: optional root directory, files outside which kustomize is not allowed to copy and localize; if not specified, + takes on value of `target` +* `newDir`: destination directory of the localized copy of `target` + +The command creates a copy of the `target` kustomization and the local files that `target` references at `newDir`. We +define the "files that `target` references" as: -The command creates a copy of `target` at `newDir` in which each kustomization file, from the top-level to any -recursively referenced, has local paths to downloaded files instead of remote references for the following kustomization -fields: +* kustomization files that `target` directly or transitively references +* configuration files that referenced kustomization files reference +* exec binaries of referenced KRM functions -* `resources` -* `components` -* `bases` -* `openapi:` -     `paths` +Here, configuration file means a non-kustomization yaml file. The command only copies referenced files that reside +inside `scope`. -A new `localized-files` directory holds the downloaded files (or directory) -at: +The command localizes the copy of `target` at `newDir` by downloading all remote files that `target` references. The +downloaded files placed in a new `localized-files` directory next to the file that referenced the downloaded files. +Inside `localized-files`, the downloads are located on path: 
 remote-host / organization / repo / version / path-to-file-in-repo
-Each `localized-files` directory is located in the same directory as the kustomization file that referenced the -downloaded files. - -To help ensure that `newDir` is a clean copy, the command overwrites every absolute path into `target` to point -to `newDir` before processing the path. +The command rewrites remote references in `newDir` to the local paths of the downloaded files. To help ensure +that `newDir` is a clean copy, the command additionally overwrites absolute path references into `target` to point +to `newDir`. **Error cases**: * `target` does not have a top-level kustomization file +* `scope` does not contain `target` * `newDir` already exists * `localized-files` directory already exists * remote reference does not have a version @@ -99,10 +101,13 @@ to `newDir` before processing the path. * cycle of kustomization file references exists **Warning cases**: -* `newDir` refers to `base` kustomization layer outside of `newDir`, and `base` has remote references -* KRM function has container image or exec binary that the user might not have locally +* `target` references a local path that traverses outside of `scope` +* KRM function has container image that the user might not have locally + +For the warning cases, the command will ignore the reference and continue execution. -If the command runs without any errors, `kustomize build` on `target` and `newDir` should produce the same output. +If the command runs without any errors or warnings, `kustomize build` without `--load-restrictor LoadRestrictionsNone` +on `target` and `newDir` should produce the same output. ### User Stories @@ -136,17 +141,17 @@ pipeline does not have external network access. Fortunately, I remember that I can run `kustomize localize` on `example/overlay` on my local machine. I can then upload the localized directory to my company’s internal package management site for the CI/CD pipeline to pull and build instead. I run -`kustomize localize example/overlay example/localized-overlay`, but I get the following error: +`kustomize localize example/overlay example/localized-overlay`, but I get the following warning: ``` $ kustomize localize example/overlay example/localized-overlay -Warning: kustomization directory example/overlay refers to remote resources in example/base +Warning: File example/overlay/kustomization.yaml refers to ../base on line 2. This reference is outside of scope: + example/overlay. kustomize localize will skip this path. ``` -because I forgot that `kustomize localize` can only localize remote references originating from within the -`target`, `example/overlay`. Therefore, command could not localize the remote references in `example/base`, -which `example/overlay/kustomization.yaml` locally references. `kustomize build example/localized-overlay` will still -run correctly outside the CI/CD pipeline, but will still require network access: +because I forgot that `kustomize localize` can only process local references to files within +`scope`. Therefore, the command could not copy `example/base` to `example/localized-overlay`or localize the remote +references in `example/base`. The resulting file structure is as follows: ``` └── example @@ -166,86 +171,101 @@ run correctly outside the CI/CD pipeline, but will still require network access: └── kustomization.yaml ``` +`kustomize build example/localized-overlay` will still run correctly outside the CI/CD pipeline because I chose to +place `example/localized-overlay` in the same directory as `example/overlay`. As a result, relative paths, +namely `../base`, in `example/localized-overlay` will point to the same files as their counterparts in `example/overlay` +. However, `kustomize build example/localized-overlay` will still require network access to run. + #### Story 2 I am back again from **Story 1**, but this time ready to localize `base` too, instead of just the `overlay` directory. -To localize both, I change my `target` argument from `example/overlay` to `example` to make a copy of both directories. -I add a top-level kustomization file to `example` with the following commands: - -``` -kustomize init; kustomize edit add resource overlay -``` +To localize both, I set my `scope` argument to `example` to make a copy of both directories. My setup still looks like +that at the end of **Story 1**: -I have deleted `example/localized-overlay` from **Story 1**. My setup now looks like this: ``` └── example - ├── kustomization.yaml ├── overlay │ └── kustomization.yaml + ├── localized-overlay + │ ├── kustomization.yaml + │ └── localized-files + │ └── github.com + │ └── kubernetes-sigs + │ └── kustomize + │ └── v1.0.6 + │ └── examples + │ └── helloWorld + │ └── deployment.yaml └── base └── kustomization.yaml ``` -``` -# example/kustomization.yaml -resources: - - ./overlay -``` -with all other files having the same contents. After I run `kustomize localize example localized-example`, I get the +After I run `kustomize localize example/overlay example new-space/localized-example`, I get the following: + ``` -├── example # the old kustomization directory -│ ├── kustomization.yaml +├── example # old kustomization directory │ ├── overlay │ │ └── kustomization.yaml +│ ├── localized-overlay +│ │ ├── kustomization.yaml +│ │ └── localized-files +│ │ └── github.com +│ │ └── kubernetes-sigs +│ │ └── kustomize +│ │ └── v1.0.6 +│ │ └── examples +│ │ └── helloWorld +│ │ └── deployment.yaml │ └── base │ └── kustomization.yaml -└── localized-example # the new, localized kustomization directory - ├── kustomization.yaml - ├── base - │ ├── kustomization.yaml - │ └── localized-files - │ └── github.com - │ └── kubernetes-sigs - │ └── kustomize - │ └── v1.0.6 - │ └── examples - │ ├── helloWorld - │ │ └── configMap.yaml - │ └── multibases - │ ├── base - │ │ └── pod.yaml - │ ├── dev - │ ├── kustomization.yaml - │ ├── production - │ └── staging - └── overlay +└── new-space + └── localized-example # the new, localized kustomization directory ├── kustomization.yaml - └── localized-files - └── github.com - └── kubernetes-sigs - └── kustomize - └── v1.0.6 - └── examples - └── helloWorld - └── deployment.yaml + ├── base + │ ├── kustomization.yaml + │ └── localized-files + │ └── github.com + │ └── kubernetes-sigs + │ └── kustomize + │ └── v1.0.6 + │ └── examples + │ ├── helloWorld + │ │ └── configMap.yaml + │ └── multibases + │ ├── base + │ │ └── pod.yaml + │ ├── dev + │ ├── kustomization.yaml + │ ├── production + │ └── staging + └── overlay + ├── kustomization.yaml + └── localized-files + └── github.com + └── kubernetes-sigs + └── kustomize + └── v1.0.6 + └── examples + └── helloWorld + └── deployment.yaml ``` ``` -# localized-example/overlay/kustomization.yaml +# new-space/localized-example/overlay/kustomization.yaml resources: - ../base - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/deployment.yaml ``` ``` -# localized-example/base/kustomization.yaml +# new-space/localized-example/base/kustomization.yaml resources: - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/multibases - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/configMap.yaml ``` -Now, I upload `localized-example` from my local setup to my company’s internal package management site. I change the -commands in my CI/CD pipeline to pull `localized-example` before running `kustomize build localized-example`, and the -command executes successfully! +Now, I upload `new-space/localized-example` from my local setup to my company’s internal package management site. I +change the commands in my CI/CD pipeline to pull `localized-example` before running `kustomize build localized-example`, +and the command executes successfully! ### Risks and Mitigations N/A @@ -266,9 +286,10 @@ different potential `target` directories cannot share copies either. ## Drawbacks -Users, like the one in the **User Stories** section, whose kustomization layers are in sibling directories need to -perform the extra step of creating a top-level kustomization file. However, many existing kustomize use cases also -require this step, and as shown in **Story 2**, users can create kustomization files relatively easily via command line. +Users whose layered kustomizations form a complex directory tree structure may have a hard time finding an +appropriate `scope`. However, many kustomizations exist in repositories, allowing the user to easily choose the repo +root as a valid `scope`. The warning messages that `kustomize localize` outputs for reference paths that extend +beyond `scope` should also help. ## Alternatives @@ -277,24 +298,34 @@ require this step, and as shown in **Story 2**, users can create kustomization f different kustomization files to share configurations. On top of that, if `kustomize build` had the added functionality to check for previous downloads of remote references at said global location, `kustomize localize` would not need to overwrite the remote references in `target` to the local downloads. As a result, `kustomize localize` - would need to neither write to `target` nor copy `target` into `newDir`. The user would not need to create a top-level - kustomization file either.

+ would need to neither write to `target` nor copy `target` into `newDir`. The user would not need to specify a `scope` + either.

Despite its advantages, the alternative design violates the self-contained nature of each kustomize layer. Users would be unable to upload a fully localized kustomization directory in version control. Furthermore, this alternative complicates the existing kustomize workflow by requiring the setup of global environment variables. +

* The command could, instead of making a copy, modify `target` directly. However, users would not have an easy way to undo the command, which is undesirable. +

+ +* Instead of requiring the user to specify a second argument `scope`, the command could by definition limit its copying + to `target`. However, in the case of **Story 1**, the command would force the user to set `target` to `example` in + order to include `example/base` in the localization of `example/overlay`. The user would then have to create a + kustomization file at `example` that points to `example/overlay` under the `resources` field. The creation of the + kustomization file solely for this purpose is messy and more work for the user. ## Rollout Plan + This command will have at least alpha and GA releases. Depending on user feedback, we may add a beta. ### Alpha -This release will limit the depth of nested `localized-files` to 1 layer. In other words, the command will ignore remote -kustomization directory references that originate from within a `localized-files` directory. In addition, this release -will ignore KRM functions. The command will not output warnings for remote KRM images or exec binaries. +This release will ignore + +* local kustomization files that `target` references +* KRM functions The entire command will be new in the alpha release, and so will not require an alpha flag. The command will not be available in `kubectl kustomize` either as kubectl only has `kustomize build` builtin. From 883be09be93832a84e8a61883ce2098e811ee130 Mon Sep 17 00:00:00 2001 From: Anna Song Date: Wed, 11 May 2022 23:44:58 -0700 Subject: [PATCH 4/4] Address feedback in comments --- proposals/22-04-localize-command.md | 270 ++++++++++++++-------------- 1 file changed, 133 insertions(+), 137 deletions(-) diff --git a/proposals/22-04-localize-command.md b/proposals/22-04-localize-command.md index 003aa8398f..2bcb13cda8 100644 --- a/proposals/22-04-localize-command.md +++ b/proposals/22-04-localize-command.md @@ -35,14 +35,20 @@ The proposed command nearly achieves the solution by downloading all remote file by a recursively referenced kustomization file. The command also downloads remote exec binaries of referenced KRM functions, which are the only potential source of remote files other than kustomizations. The only remote files that this proposal does not cover and that `kustomize build` still needs to run are remote images and custom fields in KRM -functions. Downloaded images would live only in local caches and kustomize cannot know the form that custom fields will -take. Thus, neither is worth localizing. +functions. Downloaded images would live only in local caches and thus, are not worth localizing. Kustomize cannot +currently identify custom fields, though this may change with one of the proposed solutions +in [issue #4154](https://github.com/kubernetes-sigs/kustomize/issues/4154). + +The proposed command has the added benefit of increasing user confidence in the integrity of their kustomization builds. +Locally downloaded files, unlike urls, give users full control of file content. At the same time, the command does this +without modifying the original kustomization so that users can always run the command on the original again to fetch +upstream changes. **Goals:** -1. This command should localize - * all remote files that a kustomization file directly references - * remote exec binaries of referenced KRM functions +1. This command should localize + * all remote files that a kustomization file directly references + * remote exec binaries of referenced KRM functions This command achieves this goal if, in the absence of remote images and custom fields in KRM functions, `kustomize build` can run on the localized copy without network access. @@ -58,16 +64,22 @@ take. Thus, neither is worth localizing. The command takes the following form: 
-kustomize localize target scope newDir
+kustomize localize target newDir [-s scope] [-n]
where the arguments are: * `target`: a directory with a top-level kustomization file that kustomize will localize; can be a path to a local directory or a url to a remote directory -* `scope`: optional root directory, files outside which kustomize is not allowed to copy and localize; if not specified, +* `newDir`: optional destination directory of the localized copy of `target`; if not specified, the destination is a + directory named `localized-{target}` in the same directory as `scope` + +and the flags are: +* `-s`, `--scope` + `scope`: optional root directory, files outside which kustomize is not allowed to copy and localize; if not specified, takes on value of `target` -* `newDir`: destination directory of the localized copy of `target` +* `-n`, `--no-verify`: do not verify that the outputs of `kustomize build` for `target` and `newDir` are the same after + localization The command creates a copy of the `target` kustomization and the local files that `target` references at `newDir`. We define the "files that `target` references" as: @@ -76,38 +88,58 @@ define the "files that `target` references" as: * configuration files that referenced kustomization files reference * exec binaries of referenced KRM functions -Here, configuration file means a non-kustomization yaml file. The command only copies referenced files that reside -inside `scope`. +Here, configuration file means a non-kustomization yaml file. The command cannot run on `target`s that need +the `--load-restrictor LoadRestrictionsNone` flag on `kustomize build`. The command only copies referenced files that +reside inside `scope`. -The command localizes the copy of `target` at `newDir` by downloading all remote files that `target` references. The -downloaded files placed in a new `localized-files` directory next to the file that referenced the downloaded files. -Inside `localized-files`, the downloads are located on path: +The command localizes the copy of `target` at `newDir` by downloading all remote files that `target` +references. Users do not have executable permission for downloaded exec binaries that KRM functions reference and for +each such exec binary, the command will print a warning message to that effect. + +The command creates a new `localized-files` directory, next to the file that referenced the downloaded files, to hold +said files. Inside `localized-files`, the downloads are located on path: 
-remote-host / organization / repo / version / path-to-file-in-repo
+domain / organization / repo / version / path/to/file/in/repo
-The command rewrites remote references in `newDir` to the local paths of the downloaded files. To help ensure +where `version` corresponds to the `ref` query string parameter in the url, though ideally `version` ia a stable tag as +opposed to a branch. + +The command replaces remote references in `newDir` with the local paths of the downloaded files. To help ensure that `newDir` is a clean copy, the command additionally overwrites absolute path references into `target` to point to `newDir`. +As a convenience to the user, in the absence of the `--no-verify` flag, the command automatically tries to +run `kustomize build`, without any flags, on `target` and the localized `newDir` to compare their outputs. The command +indicates success if the outputs match and throws an error with the diff summary otherwise. This check, however, is not +useful for certain `target`s, including those that need flags to build. In these cases, the command prints next steps +that users can follow to check the output themselves. For example, for `target`s that reference KRM functions with a +remote exec binary, the command suggests the user: + +1. add executable permissions for the downloaded exec binaries in `newDir` **that the user trusts** +2. run `kustomize build` with flags `--enable-alpha-plugins --enable-exec` and self-verify the outputs + **Error cases**: + * `target` does not have a top-level kustomization file -* `scope` does not contain `target` +* `kustomize build` needs `--load-restrictor LoadRestrictionsNone` to run on `target` * `newDir` already exists +* `scope` does not contain `target` +* `target` references a local path that traverses outside of `scope` +* remote reference does not have a `version` * `localized-files` directory already exists -* remote reference does not have a version * kustomization file is malformed * cycle of kustomization file references exists +* `kustomize build` produces different output for `target` and `newDir` in the absence of `--no-verify` -**Warning cases**: -* `target` references a local path that traverses outside of `scope` -* KRM function has container image that the user might not have locally +Depending on feedback, we may add an `--overwrite` flag in the future to allow users to update an existing `newDir` by +running the command again. -For the warning cases, the command will ignore the reference and continue execution. +**Warning cases**: -If the command runs without any errors or warnings, `kustomize build` without `--load-restrictor LoadRestrictionsNone` -on `target` and `newDir` should produce the same output. +* KRM function references remote exec binary, in which case the downloaded exec binary is not executable +* KRM function has container image that the user might not have locally ### User Stories @@ -115,20 +147,20 @@ on `target` and `newDir` should produce the same output. My company’s CI/CD pipeline currently pulls an `example` directory from our internal package management site. I want the CI/CD pipeline to additionally run `kustomize build example/overlay`. My setup looks like this: -``` +```shell └── example ├── overlay │ └── kustomization.yaml └── base └── kustomization.yaml ``` -``` +```shell # example/overlay/kustomization.yaml resources: - ../base - https://raw.githubusercontent.com/kubernetes-sigs/kustomize/v1.0.6/examples/helloWorld/deployment.yaml ``` -``` +```shell # example/base/kustomization.yaml resources: - github.com/kubernetes-sigs/kustomize/examples/multibases?ref=v1.0.6 @@ -141,52 +173,24 @@ pipeline does not have external network access. Fortunately, I remember that I can run `kustomize localize` on `example/overlay` on my local machine. I can then upload the localized directory to my company’s internal package management site for the CI/CD pipeline to pull and build instead. I run -`kustomize localize example/overlay example/localized-overlay`, but I get the following warning: - -``` -$ kustomize localize example/overlay example/localized-overlay -Warning: File example/overlay/kustomization.yaml refers to ../base on line 2. This reference is outside of scope: - example/overlay. kustomize localize will skip this path. -``` - -because I forgot that `kustomize localize` can only process local references to files within -`scope`. Therefore, the command could not copy `example/base` to `example/localized-overlay`or localize the remote -references in `example/base`. The resulting file structure is as follows: +`kustomize localize example/overlay -s example`, where my `target` is `example/overlay`, I accept the default location +and name of `newDir`, and I expand my `scope` to `example` because `example/overlay` references `example/base`. I get +the following output: +```shell +$ kustomize localize example/overlay -s example +SUCCESS: example/overlay, localized-overlay produce same kustomize build output ``` -└── example - ├── overlay - │ └── kustomization.yaml - ├── localized-overlay - │ ├── kustomization.yaml - │ └── localized-files - │ └── github.com - │ └── kubernetes-sigs - │ └── kustomize - │ └── v1.0.6 - │ └── examples - │ └── helloWorld - │ └── deployment.yaml - └── base - └── kustomization.yaml -``` - -`kustomize build example/localized-overlay` will still run correctly outside the CI/CD pipeline because I chose to -place `example/localized-overlay` in the same directory as `example/overlay`. As a result, relative paths, -namely `../base`, in `example/localized-overlay` will point to the same files as their counterparts in `example/overlay` -. However, `kustomize build example/localized-overlay` will still require network access to run. -#### Story 2 - -I am back again from **Story 1**, but this time ready to localize `base` too, instead of just the `overlay` directory. -To localize both, I set my `scope` argument to `example` to make a copy of both directories. My setup still looks like -that at the end of **Story 1**: - -``` -└── example - ├── overlay - │ └── kustomization.yaml - ├── localized-overlay +```shell +├── example # old kustomization directory +│ ├── overlay +│ │ └── kustomization.yaml +│ └── base +│ └── kustomization.yaml +└── localized-overlay # the new, localized kustomization directory + ├── kustomization.yaml + ├── base │ ├── kustomization.yaml │ └── localized-files │ └── github.com @@ -194,91 +198,79 @@ that at the end of **Story 1**: │ └── kustomize │ └── v1.0.6 │ └── examples - │ └── helloWorld - │ └── deployment.yaml - └── base - └── kustomization.yaml -``` - -After I run `kustomize localize example/overlay example new-space/localized-example`, I get the -following: - -``` -├── example # old kustomization directory -│ ├── overlay -│ │ └── kustomization.yaml -│ ├── localized-overlay -│ │ ├── kustomization.yaml -│ │ └── localized-files -│ │ └── github.com -│ │ └── kubernetes-sigs -│ │ └── kustomize -│ │ └── v1.0.6 -│ │ └── examples -│ │ └── helloWorld -│ │ └── deployment.yaml -│ └── base -│ └── kustomization.yaml -└── new-space - └── localized-example # the new, localized kustomization directory + │ ├── helloWorld + │ │ └── configMap.yaml + │ └── multibases + │ ├── base + │ │ └── pod.yaml + │ ├── dev + │ ├── kustomization.yaml + │ ├── production + │ └── staging + └── overlay ├── kustomization.yaml - ├── base - │ ├── kustomization.yaml - │ └── localized-files - │ └── github.com - │ └── kubernetes-sigs - │ └── kustomize - │ └── v1.0.6 - │ └── examples - │ ├── helloWorld - │ │ └── configMap.yaml - │ └── multibases - │ ├── base - │ │ └── pod.yaml - │ ├── dev - │ ├── kustomization.yaml - │ ├── production - │ └── staging - └── overlay - ├── kustomization.yaml - └── localized-files - └── github.com - └── kubernetes-sigs - └── kustomize - └── v1.0.6 - └── examples - └── helloWorld - └── deployment.yaml + └── localized-files + └── github.com + └── kubernetes-sigs + └── kustomize + └── v1.0.6 + └── examples + └── helloWorld + └── deployment.yaml ``` -``` -# new-space/localized-example/overlay/kustomization.yaml +```shell +# localized-overlay/overlay/kustomization.yaml resources: - ../base - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/deployment.yaml ``` -``` -# new-space/localized-example/base/kustomization.yaml +```shell +# localized-overlay/base/kustomization.yaml resources: - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/multibases - ./localized-files/github.com/kubernetes-sigs/kustomize/examples/helloWorld/configMap.yaml ``` -Now, I upload `new-space/localized-example` from my local setup to my company’s internal package management site. I -change the commands in my CI/CD pipeline to pull `localized-example` before running `kustomize build localized-example`, -and the command executes successfully! +Now, I upload `localized-overlay` from my local setup to my company’s internal package management site. I change the +commands in my CI/CD pipeline to pull `localized-overlay` before running `kustomize build localized-overlay`, and the +command executes successfully! ### Risks and Mitigations -N/A + +One could argue that while uploading the localized `newDir` to a repository, a user could accidentally leak Secrets that +were originally remote in a more private repo. This event is not very likely, as the servers that the user intends to +consume the localized kustomizations often only have access to internal, private networks and repos. Nonetheless, users +should have enough context in the case of Secrets to make the right decision. Users should have a basic understanding of +the files that their `target` kustomizations reference and of the files that they plan to upload to repos. Secret +configurations are also not too difficult to identify. + +Exec binaries that KRM functions reference are a different story. `kustomize localize` downloads remote exec binaries +that, if malicious, are capable of almost anything during subsequent `kustomize build` calls. The command mitigates this +risk by leaving these downloaded exec binaries without executable permissions and warning the user, as mentioned in +**Proposal**. `kustomize build` can only run the exec binary after the user deems the binary safe and changes its +permissions. + +Still another risk may be that if a user's kustomization tree is large, `kustomize localize` may be copying files from +unexpected locations. The command mitigates this risk with the `scope` flag. If not set, `kustomize localize` only +copies files in `target`. Otherwise, the user specifies `scope`and understands that `kustomize localize` only copies +files in `scope`. The qualification that the command can only localize `target`s that follow load restrictions helps +mitigate this risk as well. ### Dependencies + N/A ### Scalability -A chain of remote kustomization directories in which the current kustomization file references the next remote -kustomization directories could create a `newDir` with deeply nested `localized-files`. This directory structure would -impede users’ navigation of `newDir`. However, this scenario should be unlikely as most kustomizations only consist of a -few layers. +Large kustomization trees slows the performance of `kustomize localize`. These trees can have large local subtrees, have +large remote subtrees, be deeply nested, or be wide, with each overlay referencing multiple bases. Regardless of the +cause, large kustomization trees inevitably takes longer to copy and download. Parts of the kustomize code are not +thread-safe, which precludes parallel execution. + +On a separate note, a chain of remote kustomization directories in which the current kustomization file references the +next remote kustomization directories could create a `newDir` with deeply nested `localized-files`. This directory +structure would impede users’ navigation of `newDir`. However, this scenario should be unlikely as most kustomizations +only consist of a few layers. The creation of the `localized-files` directory local to the referencing kustomization file additionally prevents the different layers of kustomization files from sharing the same copy of the remote files. Following the same logic, @@ -322,10 +314,14 @@ This command will have at least alpha and GA releases. Depending on user feedbac ### Alpha -This release will ignore +This release will not support -* local kustomization files that `target` references +* load restrictions check * KRM functions +* absolute paths +* `target` kustomization with reference cycles +* any verification in the form of `--no-verification` flag or automatically running `kustomize build` at the end of + localization The entire command will be new in the alpha release, and so will not require an alpha flag. The command will not be available in `kubectl kustomize` either as kubectl only has `kustomize build` builtin.