kustomize v4 high-impact breaking changes

[stefanprodan]

The latest release of kustomize comes with breaking changes that have a high impact on Flux users.

❗ TLDR: Do not upgrade to v4, stay on v3.9.3 until we figure out a way forward.

Removal of go-getter

The hashicorp/go-getter package has been dropped from kustomize due to go-getter's license not being on the CNCF approved LICENSE list.

Ref: kubernetes-sigs/kustomize#3578

Impact

The set of URLs accepted by kustomize in the resources filed is reduced to only file system paths or values compatible with git clone. This means you can no longer use resources from archives (zip, tgz, etc) nor S3, GCS, Minio, Mercurial, etc.

💥 This means you can no longer install Flux with kustomize build github.com/fluxcd/flux2/manifests/install?ref=v0.8.0 unless you use kustomize v3.9.3.

$ kustomize version 
{Version:kustomize/v4.0.1 GitCommit:516ff1fa56040adc0173ff6ece66350eb4ed78a9 BuildDate:2021-02-14T05:00:02+00:00 GoOs:darwin GoArch:amd64}

$ kustomize build github.com/fluxcd/flux2/manifests/install?ref=v0.8.0
Error: accumulating resources: accumulation err='accumulating resources from '../bases/source-controller': '/private/var/folders/bh/bp5hplld7g915wdvkt5w0y9w0000gn/T/kustomize-040141028/manifests/bases/source-controller' must resolve to a file': recursed accumulation of path '/private/var/folders/bh/bp5hplld7g915wdvkt5w0y9w0000gn/T/kustomize-040141028/manifests/bases/source-controller': accumulating resources: accumulation err='accumulating resources from 'https://github.com/fluxcd/source-controller/archive/v0.8.0.zip//source-controller-0.8.0/config/crd': yaml: line 151: mapping values are not allowed in this context': evalsymlink failure on '/private/var/folders/bh/bp5hplld7g915wdvkt5w0y9w0000gn/T/kustomize-247279347/archive/v0.8.0.zip/source-controller-0.8.0/config/crd' : lstat /private/var/folders/bh/bp5hplld7g915wdvkt5w0y9w0000gn/T/kustomize-247279347/archive: no such file or directory

✔️ Flux users should use flux install --export instead of kustomize build.

💥 Once we upgrade kustomize-controller to kustomize/api v0.8, Flux users can no longer make use of GitHub releases URLs as remote resources due to zip and tar.gz not being supported.

💡 The Flux team is considering offering a solution for defining sources from GitHub releases and remote archives.

Removal of k8s.io/apimachinery

The dependencies on k8s.io packages have been dropped to update the kustomize functionality built into kubectl.

Ref: kubernetes-sigs/kustomize#2506

Impact

Once we upgrade kustomize-controller to kustomize/api v0.8, Flux users will be affected by critical bugs present in kustomize/kyaml library:

• 💥 If you have duplicate keys in your manifests, the kustomize-controller will crash loop with no option of recovery, ref: kyaml panics on duplicate keys kubernetes-sigs/kustomize#3480
• 💥 If you have non-string keys in your manifests, the kustomize-controller will not be able to reconcile your clusters, ref: Support for integer keys kubernetes-sigs/kustomize#3446

💡 The Flux team will withhold from upgrading kustomize/api to v0.8 for some time, this means other bug fixes in kustomize will not be available to Flux, so we can't keep this up for long.

Renaming of flags

The kustomize flags that contain underscores have been replaced with hyphens to be compatible with kubectl.

Ref: kubernetes-sigs/kustomize#3601

Impact

💥 This means Flux docs and the CI validation scripts the we've provided to users will no longer work.

💡 Once we find a way to migrate Flux to kustomize/api v0.8, we'll update the documentation and the example scripts.

[o6uoq]

There's an issue with brew and re-installing to a later version e.g. 3.9.3

Work Around

• brew extract --version=3.9.3 kustomize homebrew/cask

There will be a file named something like /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-cask/Formula/kustomize@3.9.3.rb, copy and paste this file and path and run brew install:

• brew install /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/homebrew/homebrew-cask/Formula/kustomize@3.9.3.rb

Confirm kustomize has installed to the pinned version of 3.9.3:

❯ kustomize version
{Version:kustomize/v3.9.3 GitCommit:1ae8303bdc9372bc7c15942df6e9cf5d67fdba1a BuildDate:2021-02-15T18:11:38+00:00 GoOs:linux GoArch:amd64}

[stefanprodan]

Starting with flux v0.9, the Kustomize overlays under manifests/bases are compatible with v4.

Generating the Flux install manifests with the latest kustomize CLI is working again:

kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=v0.9.1

[stefanprodan]

One major blocker has been fixed, kyaml/v0.10.14 no longer panics on duplicate keys, with this we should be able to upgrade kustomize-controller to sigs.k8s.io/kustomize/api v0.8.5.

The only outstanding issue left is kubernetes-sigs/kustomize#3446

[stefanprodan]

In other news, kubectl kustomize has been updated to v4 and it will be available in Kubernetes 1.21.

[aashrit93]

Hi @stefanprodan ,
trust you are doing well.

I require your guidance on an issue i am facing with using kustomize and flux.
Issue details below :
The usecase is quite simple , its to have a kustomize setup where the
The overlays repo refers to another repository that contains the 'bases'

if i use kubectl apply -k or kustomize build i see the expected output
however when i try to use the same kustomize file with gitops ( setup using flux install, flux create source and flux create kustomization ) i see the below error during flux kustomization creation

• accumulateFile error: "accumulating resources from 'https://aashrit93@github.com/aashrit93/kustomize_bases.git/bases/helloworld': YAML file [https://aashrit93@github.com/aashrit93/kustomize_bases.git/bases/helloworld] encounters a format error.\nerror converting YAML to JSON: yaml: line 159: mapping values are not allowed in this context\n"
• loader.New error: "git cmd = '/usr/bin/git fetch --depth=1 origin HEAD': exit status 128"

Kubectl version 1.19.7
kustomize version v4.0.5
flux version 0.13.2

Any help is appreciated , thanks in advance

[yebyen]

@aashrit93 Please re-read the same issue that you've commented on asking for help. Kustomize v4 cannot be used with Flux.

Downgrade your Kustomize CLI to match the 3.x major series requirement explained here. The last release in the 3.x series (for now?) is 3.10.0. This is the version that Flux's Kustomize controller uses, to get the same behavior from your Kustomize CLI, you should read and follow the other advice in the FAQ: https://fluxcd.io/docs/faq/#what-is-the-behavior-of-kustomize-used-by-flux

[stefanprodan]

We've published an image of kustomize-controller compatible with kustomize v4, please give it try and report all issues to this PR fluxcd/kustomize-controller#343

To test this please use the following image:

ghcr.io/fluxcd/kustomize-controller:kustomize-v4-rc-de0310e7

To upgrade kustomize-controller on an cluster bootstrapped with flux, add the following images section to your flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
images:
- name: ghcr.io/fluxcd/kustomize-controller
  newName: ghcr.io/fluxcd/kustomize-controller
  newTag: kustomize-v4-rc-de0310e7

[aashankar]

thanks @yebyen and @stefanprodan for taking the time to look into this , much appreciated. Will try the above config , this helps a lot !

[rajivchirania]

One major blocker has been fixed, kyaml/v0.10.14 no longer panics on duplicate keys, with this we should be able to upgrade kustomize-controller to sigs.k8s.io/kustomize/api v0.8.5.

The only outstanding issue left is kubernetes-sigs/kustomize#3446

@stefanprodan With the latest flux version 0.15.3 installed via terraform i am still getting this error when helmrelease are trying to render
https://github.com/fluxcd/flux2/issues/1566