kustomize vars - enhance or replace?

[monopole]

Kustomize has a vars feature that shipped early in the
project. The project's view of its purpose and direction has
focussed since then, and the feature now appears as a outlier in
the kustomize feature set for reasons discussed below.

The vars feature is also a common topic in issues being filed,
not necessarily because of actual bugs but because vars appear to
be template variables, but don't function as such.

This issue is a parent issue to gather var related issues,
identify common problems and consider solutions to these problems
that don't require vars.

What’s a var?

A var in kustomize is a reflection mechanism, allowing a value
defined in one YAML configuration field (e.g. an IP address) to
be copied to other locations in the YAML. It has a source spec
and any number of targets.

• source spec: a field in a kustomization.yaml
  file associating an uppercase var name like
  VAR with a specific field in a specific resource
  instance, e.g. the image name in the Deployment named
  production. This field is the source of the var’s
  value.

• targets: instances of the string $VAR in resource
  instance fields, identifying where to put the var’s
  value. The placement of $VAR is constrained to a particular
  set of fields in a particular set of resources.

It’s a don’t-repeat-yourself feature. The overall effect is
similar to the reflection provided by YAML anchors, except that
kustomize manages it up the overlay stack.

Isn't this templating?

A kustomize var smells like a template variable, but stops short
of full templating.

Full templating, with a distinct key:value (KV) file, has
drawbacks.

• A template isn't YAML; it must be rendered to make YAML.

  Kustomize allows a $VAR only in a limited set of string
  fields like container command arguments (similar to the
  contraints of the downward API), so the source material
  remains usable with generic YAML tools.

• KV files become a template-driven API wrapping the real API.

  This is fine in simple cases, but an API emerging from
  templates scales poorly to real world production setups -
  large environments, disparate configuration owners, etc.
  Kustomize vars avoid the distinct KV file by requiring
  reflection.

• The ugliness arising from shared templates - everything gets
  parameterized.

• The difficulty of rebasing templates to capture upstream
  changes into your template fork.

These drawbacks (and others) are discussed in more detail in
Brian Grant's Declarative application management in
Kubernetes.

Kustomize vars directly avoid the first two, and help avoid other
problems by simply not being the core means to generate and
customize configuration.

Kustomize vars, however, share one glaring flaw with template
variables. Their use makes the raw configuration data unusable
in an apply operation - the config data must be passed through
kustomize first before being applied.

This violates an explicit goal of kustomize; provide a means to
manipulate configuration data without making the raw
configuration unusable by kubernetes. kustomize vars would not
now be accepted as a new feature in their current form.

Issue Survey

The following attempts to capture and categorize issues related
to kustomize vars.

I want to put $VAR in some (currently disallowed) field

kustomize doesn't allow unstructured $VAR placement. A $VAR
must go into a field, and only into a particular set of fields.

Kustomize vars are handled by the var transformer. Like all
other builtin transformers, it has a builtin configuration, in
its case defined in the file varreference.go. This file
defines where $VAR can be placed in configuration yaml.

One can use the Configurations field in any kustomization
file (see this test, this other test and these
examples) to specify a file containing a custom set of field
specs in the same format as varreference.go. This allows a user
to add a $VAR to more string fields without changing kustomize
code.

The var transformer is a singleton with global scope

Kustomize plugin transformers have no effect beyond the
kustomization directory tree in whose root they are declared.
Further, in one kustomization directory one may use many
different instances of the same transformer - e.g. one can
declare multiple label transformers that add different labels to
different objects.

The var transformer, however, is special - it's a singleton with
global scope. When var transformer configuration data is
declared in a kustomization file, it's not immediately used, and
is instead merged with any var transformer configuration data
already read. The last step of a kustomize build is to take
all accumulated var configuration, build a singleton var
transformer, and run it over all objects in view.

The upshot is that one can put var definitions and var
transformer configuration anywhere (in any kustomization.yaml
file reachable from the kustomization file targetted by a
kustomize build command) and get the same global effect.

This global behavior came from early requests to have var
definitions propagate up, so that overlays could usefully decare
$VARs in their patches.

Some issues

• Variable substitution only partly working on Jobs #1782 Variable substitution only partly working on Jobs
• well-defined vars that were never replaced: EXTERNAL_IP #1734 well-defined vars that were never replaced
• Kustomize vars not applied to Namespace resources #1713 Kustomize vars not applied to Namespace resources
• Can't use kustomize vars in images #1592 Can't use kustomize vars in images
• var replacement doesn't occur on commonLabels #1585 var replacement doesn't occur on commonLabels
• Using overlay-provided secrets in a base #1553 Using overlay-provided secrets in a base
• Var reference does not work in volumes #1540 Var reference does not work in volumes

I want to define vars that aren’t strings

It so happens that the current implementation of vars only allows
strings, because only string fields (container command line args
and mount paths) are allowed by default, and the yaml libraries
in use make a distinction between fields of string, number, map,
etc.

Relaxing this is straightforward (a bit more code, tests, and
error handling) but pointless if vars are deprecated.

Some issues

• Variable is expected to be string #1721 Variable is expected to be string

I want to use diamonds

Suppose one has a top level overlay, called all, that merges
sibling levels dev, staging, prod (variants) by specifying
them as resources - i.e. the file all/kustomization.yaml
contains:

resources:
- ../prod
- ../staging
- ../dev

Suppose in turn that these variants modify a common base. A var
defined in that common base will be defined three times at the
all level, a currently disallowed behavior.

This is analogous to the compiler rejecting a construct like

type Bar struct {
  someVar int
}
type Foo struct {
  Bar
  Bar
}

@tkellen proposes a fix for this, #1620, allowing repeatedly
defined var names as long as the source of the underlying value
can be shown to be the same.

The solution works for particular use cases, but leaves us with a
global vars that will sometimes work and sometimes not work.
When they don't work - when the values don't match - what should
the error encourage the user to do? Rename/re-arrange vars? Or
do something else entirely?

If vars are retained, and get more complex scopes, then there
should be a clear design associated with them. We don't actually
need that design to know that it would be a firm step into
configuration language territory, a non-goal of kustomize.

Meta question: why make a diamond structure?

Possibly to do one-shot apply operations. With the above
kustomization file, one can deploy all environments with one
command:

result=$(kustomize build all | kubectl apply -f -)

The price of this convenience is a completely useless result from
apply, analogous to conflating the purchase of real estate, a car
and a sandwich to one financial transaction, and accepting that
it must all be undone because the sandwich had onions you didn't
order.

A better way to do this as one command is make a scripted loop
which can analyze the apply result for each environment:

for env in 'prod staging dev'; do
   result=$(kustomize build $env | kubectl apply -f -)
   handle(result)
done

Some issues

• don't cause variable conflicts when values are the same #1600 don't cause variable conflicts when values are the
  same
• Error when composing several identical bases that use the same var: "var ... already encountered" #1248 Error when composing several identical bases that use the same var: "var ... already encountered

I just want a simple KV file

Here a user wants to get the value for the vars from some
external KV file, rather than reflexively from some other part of
the config.

One reason not to do this is that there's already a mechanism to
convert KV files and other data representations into
configmaps, and vars already know how to
source configmaps, so let's try that first. The purpose of
configmaps is to hold miscellaneous config data intended for
container shell variables and command line arguments; they're
aligned with the intent of kustomize vars. Having the configmaps
in the cluster has other advantages, and no downside outside a
questionable concern that configmaps will forever accumulate in
storage.

Another reason not to do this is it would place kustomize firmly
in the template business, which is a non-goal of kustomize.
Kustomize users that also really want to use some templating can
do so using some other tool - sed, jinja, erb, envsubst, kafka,
helm, ksonnet, etc. - as the first stage in configuration
rendering.

It's possible to use a kustomize generator plugin to drive such a
tool (e.g. sed example, helm example) in this first stage,
hopefully as a first step in coverting away from template use
entirely.

Some issues

• Enhancement: Literal vars #318 Enhancement: Literal vars
• Support for variable replacement in literals #1737 Support for variable replacement in literals

Summary

Kustomize vars feel incomplete from from a template-user point of
view. The existence of this feature inspires an urge to
generalize vars to provide full templating and/or configuration
language-like variable scoping.

Alternatives to vars

The kustomize approach is to eschew the temptation to generalize
to yet another config language or templating tool, and solve
specific kuberenetes configuration problems with a dedicated
transformer or generator that, itself, is easy to configure and
use.

Vars have been a sort of stopgap, used when one cannot frame the
problem as solvable by an existing transformer or generator
plugin.

What's a plugin?

Kustomize is just a finder and runner of transformer and
generator plugins.

Transformers

Two common uses of template variables in kubernetes is to set the
image name in a container spec, and set labels on objects.

In kustomize, these particular tasks are done with transformer
plugins - respectively, the image transformer and the label
transformer.

A transformer plugin is a chunk of code written in any language that

• is configurable by a kubernetes style YAML file,
• can be found and run by kustomize when the path to its config
  file appears in the transformers: field
  of a kustomization file,
• when run, accepts, modifies and emits YAML.

Transformers understand the structure of what they modify; they
don't need template variable placeholders to do their job.

If a use case feels like adding a field or changing the value of
a field, the kustomize way to do it is write a transformer plugin
to perform the field addition or mutation

Generators

Since vars let one copy data from one field to another, it might
be best to simply make the object or set of objects with these
fields set to the same value from the outset.

In kustomize, the way to make objects (instead of reading them
from storage) is to use a generator plugin. It's a chunk of code
written in any language that

• is configurable by a kubernetes-style YAML config file,
• can be found and run by kustomize when the path to its config
  file appears in the generators: field
  in a kustomization file,
• when run, emits kubernetes-style YAML.

A use case for generators is kubernetes Secret generation.
There's a builtin secret generator plugin, and the
documentation has an example of a plugin that
produces Secrets from a database.

A plugin's config file holds values that would otherwise appear
as part of a KV file. In the template world, it's possible to
define keys as JSON paths (YAML being a superset of JSON) so that
the KV file looks like a YAML object. But there's
no object here in the traditional sense. The template user must
rely on the branching and looping constructs of the template to
do anything other than replace keys with these values.

A kustomize generator should have documentation and unit tests -
it's a tangible, testable factory.

custom resources

While mentioning that KV files can be viewed as objects, it's
appropriate to mention custom resources.

Given a set of related, live kubernetes objects - how can they be
instantiated as one meaningful, monitorable thing with a life of
its own? Custom resources are the answer to this
question, and the general answer to extending the kubernetes API
with more objects.

Kubebuilder is a tool to help write the controller
that accompanies a custom resource definition in the kubernetes
control plane. Like any other kubernetes API YAML, custom
resource configuration can be generated or transformed by
kustomize.

Specific alternatives to vars

Replacement Transformer

This is a general purpose var replacement, coded up as an example
transformer in #1631.

The idea here is simple: eliminate embedding $VAR as the means
of defining the target, replacing target specification with
kind/field addresses in the kustomization file (as is already
done for sources).

This keeps the raw material usable - no sprinkling of dollar
variables throughout the config.

There’s a loss of functionality though. In the existing vars
implementation, the $VAR can be embedded in a container command
line string. I.e., one can replace part of a string field.

The atomic unit of replacement of this transformer is the whole
field. That said, if your command line is a list of arguments
(instead of one blank-delimitted string), the replacements
transformer should be able to target list entries by index.

Path Transformer plugin

This doesn't exist yet, but someone could write it.

One use of vars is to serve as a placeholder in a string field
that contains a file system or URL path,
e.g. /top/config/$ENV/$SUBDIR/data. The var feature would
replace $ENV and $SUBDIR with some values to resolve to a
real path.

There's no need for a var here. Kustomize has a prefix/suffix
transformer is dedicated to name field transformation. It could
be copied and adapted to editting fields known to contain paths -
prepending, postfixing or otherwise changing path declarations.

[tkellen]

Thank you for writing this up @monopole! My early vote is rip them out ASAP. I'll share a much more detailed justification with examples soon.

[monopole]

Thanks @tkellen , i'm hoping people will own some new transformers to allow that :)

[tkellen]

I'm just going to start sharing stuff incrementally or I'll never say anything.

Before we even talk about vars, I think we need to agree or disagree on the veracity of this point:

Kustomization files should be black boxes where we can do whatever we want. What comes out the other end must be raw yaml with no context about what happened inside.

If you agree, vars as implemented need to go away. Also, unless I am mistaken, agreeing also cleanly allows the "one shot" or "diamond" approach you've outlined as being a bad practice.

---

Here is a kustomization layout I have been using that supports every use case I have thrown at it for nearly a year (it relies on vars-type functionality in @jbrette's fork of kustomize but I believe it can work without it). As far as I can tell, if the features I'll write about later were added to kustomize we would have the base primitives needed to say k8s configuration management is a solved problem (speaking purely technically, this doesn't cover the sticky issues of trying to have humans collaborate together effectively that @pwittrock is trying to solve with #1962 type things).

                                                                                                                                                                                                                                                         
                                                                                                                                                                                                   one-shot dev                             +---------+  
                                                                                                                                                                                                  +-------------------+                     | dev     |  
     ENVIRONMENT: dev                                                                                                                              -----------------------------------------------| kustomization.yml |---------------------| k8s     |  
                                                                                                                                                  /contextless yaml                             / +-------------------+ contextless yaml    | cluster |  
          region.txt -----\                                                              BASE DEPLOY: dev                                        /                                             /                                            +---------+  
        env_name.txt ------\ +-------------------+                                       +-------------------+                                  /                                             /                                                          
        hostname.txt --------| kustomization.yml |---------------------------------------| kustomization.yml |---------------------------------/------------                                 /                                                           
        rds_host.txt ------/ +-------------------+ contextless yaml                    / +-------------------+\contextless yaml               /             \                               /                                                            
    project_name.txt -----/      configmap                                            /                        \                             /               \                             /                                                             
                                 generator                                           /                          \                           /                 \                           /                                                              
                                                                                    /                            \   service foo in dev    /                   \   service bar in dev    /                                                               
                                                                                   /                              \ +-------------------+ /                     \ +-------------------+ /                                                                
                STANDARD DEPLOY                                                   /                                -| kustomization.yml |/                       -| kustomization.yml |/                                                                 
                                                                                 /              APP: service-foo  / +-------------------+     APP: service-bar  / +-------------------+                                                                  
             serviceaccount.yml -----\                                          /               .                /                            .                /                                                                                         
                       role.yml ------\ +-------------------+                  /                ├── Makefile    /                             ├── Makefile    /                                                                                          
                rolebinding.yml --------| kustomization.yml |------------------                 ├── manifest.yml                              ├── manifest.yml                                                                                           
                 deployment.yml ------/ +-------------------+ contextless yaml \                └── src         \                             └── src         \                                                                                          
                    service.yml -----/                                          \                   └── main.go  \   service foo in prod          └── main.go  \   service bar in prod                                                                   
                                                                                 \                                \ +-------------------+                       \ +-------------------+                                                                  
                                                                                  \                                -| kustomization.yml |                        -| kustomization.yml |\                                                                 
                                                                                   \                              / +-------------------+\                      / +-------------------+ \                                                                
    ENVIRONMENT: prod                                                               \                            /                        \                    /                         \                                                               
                                                                                     \                          /                          \                  /                           \                                                              
          region.txt -----\                                                           \  BASE DEPLOY: prod     /                            \                /                             \                                                             
        env_name.txt ------\ +-------------------+                                     \ +-------------------+/                              \              /                               \                                                            
        hostname.txt --------| kustomization.yml |---------------------------------------| kustomization.yml |--------------------------------\-------------                                 \                                                           
        rds_host.txt ------/ +-------------------+ contextless yaml                      +-------------------+ contextless yaml                \                                              \                                                          
    project_name.txt -----/      configmap                                                                                                      \                                              \   one-shot prod                            +---------+  
                                 generator                                                                                                       \                                              \ +-------------------+                     | prod    |  
                                                                                                                                                  ------------------------------------------------| kustomization.yml |---------------------| k8s     |  
                                                                                                                                                   contextless yaml                               +-------------------+ contextless yaml    | cluster |  
                                                                                                                                                                                                                                            +---------+                                                                                                                                                                                                                                              ```

I am also mocking up a functional repository that shows this pattern here:
https://github.com/scaleoutllc/monorepo-template

...more as this evolves.

[tkellen]

Thanks @tkellen , i'm hoping people will own some new transformers to allow that :)

I understand and respect that hope. The lack of engagement between the maintainers of this project and would-be-contributors that I have watched over the last two years all but ensures I will not be contributing directly to this project unless I am made a maintainer or something drastic changes. At the moment I am hopeful the ideas I share will be powerful enough to evoke action.

[monopole]

I brought up diamonds above because their use surfaced problems with the global vars model. I noted that using a diamond solely as convenience to push to multiple environments as a one-shot unnecessarily complicates failures and retries - that's just a generality, i probably shouldn't have brought it up.

Diamonds without vars are allowed, and work:

• https://github.com/kubernetes-sigs/kustomize/blob/master/api/krusty/diamonds_test.go
• https://github.com/kubernetes-sigs/kustomize/blob/master/api/krusty/diamondcomposition_test.go

[monopole]

@tkellen As for maintainers not being engaged enough, I agree. Sounds like you're on board with killing vars. Assuming someone else isn't passionate in the other direction, we need a deprecation plan - dates, new transformers, and doc about to convert use cases. Do you want to own that?

[monopole]

Sorry - doing other things. W/r to the black box comment:

kustomization files should be black boxes where we can do whatever we want. What comes out the other end must be raw yaml with no context about what happened inside.

There are two exceptions breaking this model by relying on global data

• the var transformer (discussed ad nauseam above - no need to cover here)
• name modifiers - currently the prefixsuffixtransformer and the hashtransformer.

Name modifiers break name references (e.g. a deployment referring to a configmap).

These references are currently repaired globally, not on a per directory basis. This allows a user to specify a patch to a resource in an overlay without having to know that the resource's name had been changed in a lower level kustomization.

If one allows this convenience to the user, then knowledge of the resource's current name (possibly changing as one goes up the stack) and original name must be retained to fix references.

Currently the original name is retained in global data, for a final pass to fix references. This could be changed - one could fix references at each level, so each level emits "correct" yaml. But the original names must be retained going up. If not in related kustomize data structures, they could be slapped into the yaml itself, either as a special comment on the name field, or as a newly created secondary name field - e.g. originalName (you'd have to be sure it didn't collide with an existing field). The name reference fixer would need to know where to find these original names.

All other builtin transformers are closed with respect to the kustomization directory root - i.e. the box is black for them.

[le-duane]

Could someone suggest how to accumulate variable values down the overlay tree and apply at the last or a particular overlay using plugins?

Seems like a lot of people have differing opinions about what Kustomize should do, but for me it's about reducing duplication via overlays. Not having some kind of cohesive way to use variables would mean separating things into separate kustomizations with hard-coded values.

[monopole]

@le-duane can you be really, really specific about the use case?

[le-duane]

Thanks for your interest @monopole

I'm dealing with stateful apps, so maybe different than what most are used to.

Still a WIP, but basic overlay tree looks something like this:

           base
             |
            env
             |
        kube-cluster
             |
         db-cluster
             |
      db-cluster-nodes

Example with 2 separate sandbox DB clusters:

          cool-db
             |
          sandbox
             |
         us-west-1
          /     \
       sxb1     sbx2
        |        |
     rand-az    az-a

Example with 1 prod DB cluster spread across regions/kube clusters:

          cool-db
             |
         production
           /     \ 
     us-east-1 us-east-2
         |         |
       prd1      prd1
         |       /  \
       az-a   az-a az-c

Two things I use Vars for.

1. There are services and endpoints which must be named exactly the same as the db cluster-name. The cluster-name can be changed at layer 4 to accommodate e.g. multiple sandbox clusters for testing. This is not compatible with suffixing, because there is nothing to suffix. Therefore, they must be named with a variable that points the cluster-name label.

apiVersion: v1
kind: Service
metadata:
  name: $(CLUSTER_NAME)

2. I want to be able to change labels for AZ/Cluster-Name/Region in an overlay and have that propagate to all of the related object properties.

e.g. For adding db nodes in availability zone B, the only thing that changes in the overlay is:

apiVersion: builtin
kind: LabelTransformer
metadata:
  name: "zoneLabeler"
labels:
  az: "b"
fieldSpecs:
# StatefulSet
- path: metadata/labels
  create: true
  kind: StatefulSet
- path: spec/selector/matchLabels
  create: true
  kind: StatefulSet
- path: spec/template/metadata/labels
  create: true
  kind: StatefulSet

and the rest is configured from a variable pointing to that label e.g:

apiVersion: builtin
kind: PrefixSuffixTransformer
metadata:
  name: "instance-namer"
suffix: -$(AZ)
fieldSpecs:
- path: metadata/name
  kind: StatefulSet
- path: metadata/name
  kind: ConfigMap
- path: metadata/name
  kind: Secret

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: failure-domain.beta.kubernetes.io/zone
                operator: In
                values:
                - $(REGION)$(AZ)

# not important to this discussion, but storage is tied to an AZ to help the cluster-autoscaler make decisions when we want replicas tied to an AZ
volumeClaimTemplates:
- metadata:
    name: cool-storage
spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
        storage: 100Gi
    storageClassName: standard-$(REGION)$(AZ)

These object definitions above are all in different overlays.

It is important for us to avoid having to re-template or manually modify the same thing in multiple places any time we want to add a new database cluster or set of nodes for an existing cluster.

I think I could use a transformer plugin to template at the last mile, but that requires duplicating Vars in every db-node overlay, which seems clunky and error prone because one has to pick which vars need to change between overlays and leave the rest unchanged.
If I were to template at each overlay, so as to avoid duplicating top level vars, then I couldn't template object names because there would be a mismatch between the names at different layers. Also I couldn't override variables further down the chain.

I 100% agree other templating engines are always going to be better at that piece, but they have no knowledge of the kustomization hierarchy, which makes using them difficult from my point of view.

Finally, since diamonds were mentioned, I tried tying all resources for a kubernetes cluster together in a separate kustomization, but it blew up on the duplicate services/endpoints I mentioned earlier. If somehow duplicates could be squashed, that would be really helpful from a user and CI/CD standpoint.

edit:
Forgot to add, if it weren't for the issues with diamonds, I think I could lose a layer of duplication.

           base                     base
             |                       |    
            env                  kube-cluster
             |                      /
         db-cluster                /
             |                    /
      db-cluster-nodes  <--------|

[tkellen]

@monopole can you help me understand what you mean precisely by:

Name modifiers break name references (e.g. a deployment referring to a configmap).

If a kustomization changes the name of something I think it is a fair expectation of the consumer of the yaml stream it produces to know what the "new" name is and use it. It is very confusing to me that I could run kustomize build in a folder, observe the output, and then in some other kustomization consume what I expect to be that output and then match on something that wasn't shown (to apply a patch, for example).

Perhaps I am unintentionally relying on this critical behavior in some fashion but based on my limited understanding of the implementation it seems to me that this, like vars, should be entirely removed, or, at the very least, the "hidden" context should appear in the output yaml as you describe.

[tkellen]

@le-duane I try to use kustomize in the same manner you do, to eliminate duplication through the use of overlays. As for how to propagate values through kustomize without vars, see the very first stage in my pipeline where I generate a configmap from flat files on disk? That configmap is present in every consumer and those values should be picked up and placed in other fields through replacements (#1594). The primary missing functionality with replacements is the ability to interpolate multiple values and literal values.

[tkellen]

@le-duane, the removal of vars would mean you could no longer put $(whatever) in your source yaml. This is the basic antipattern the entire kustomize project is founded on resolving.

Here is how to accomplish one of the workflows you described without vars (this assumes you are using a layout like mine where you bring "variables" in as a configmap and pass it around wherever you need it):

This:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: failure-domain.beta.kubernetes.io/zone
                operator: In
                values:
                - $(REGION)$(AZ)

...becomes this:

replacements:
  - from:
      value: 
        - configmap:environment.data.region
        - configmap:environment.data.az
      join: ""
    to:
      target:
        kind: Deployment
      fieldrefs:
        - spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions.values[0]

Importantly, the replacement syntax I've shown above is not yet implemented. It allows multiple values to be sourced with the assumption you're going to join them into a single string on a delimiter, in your case, no delimiter.

[tkellen]

@monopole Looks like I missed your edit of this during the initial posting:

@tkellen As for maintainers not being engaged enough, I agree. Sounds like you're on board with killing vars. Assuming someone else isn't passionate in the other direction, we need a deprecation plan - dates, new transformers, and doc about to convert use cases. Do you want to own that?

I'm interested in owning this, yes, though I cannot do it in my spare time. I will be available for contract work in the near term. Is there a more direct avenue to engage in a conversation about that?

[bkuschel]

Please don't remove vars until such time the replacement transformer is able to do partial string substitution effectively without imposing base resource requirements or requiring repetitive replacements.

Please see prefix example here: #1738

Ripping it out would break us completely, my suggestions is to create a transformer out of it instead.

[tkellen]

Depending on the size of your team I would highly recommend switching to an envsubst or similar pattern ASAP. It seems clear that vars in their current form are not meant to be used in the manner you are using them and no matter what there will be breakages down the line (unless the codebase is drastically altered to respect configuration file versions).

For more context, I'm in the situation of relying on a fork to do more or less what you've done and I've taught an entire team of people how to use a feature that never landed, and, to some degree, to grok semantics that make no sense. Unwinding this is a nontrivial task.

[bkuschel]

I think we are using what it is intended for, at least according to @pwittrock 's description during the brainstorm session.
The general problem is that resources are more than k8s schemas, they contain strings, these string contain references to attributes in other kubernetes resources that change during the course of manifest generation and need to be replaced.
Patches are not a good solution for replacing a substring in a random string coming from a base as the patch would need the entire string or arrays, not to mention the problems with SMPs with CRDs and arrays (like args).
Prefix and namespaces would require an incredible amount of patches for 30+ micrososervices. With VARS (or a good replacement alternative) this is a no-brainer.

envsubst is something that would need to be done after a manifest generation so it's possible but problematic as kustomize is not generating a working manifest. Seems like an anti-pattern.

[tkellen]

Thanks for the clarification @bkuschel. I believe replacements can solve the issue you've raised RE: replacing substrings in a random string, but the configuration format and implementation most assuredly doesn't exist for it yet.

[bkuschel]

Right, this is why i'm against ripping out VARS at this time.

[ricochet]

While I agree that removing VARS is the right call longterm, I also want to echo @bkuschel in that removing VARS without a clear onboarding alternative (available at HEAD) is detrimental to consumers. I recommend starting with a deprecation warning and warnings throughout the doc to signal that a drop is incoming.

Converting a real OSS consumer like kubeflow will give the community far more legs than suggestions in issues. I reference kubeflow's KustomizeBestPractices for my own work.

Something else to consider is to add a similar implementation in the kubectl doc's
app composition and deployment guide

                      +---------------+
                      |Deployment Base|
                      +---^--------^--+
                          |        |
                +---------+-+    +-+---------+
                |Golang Base|    |Spring Base|
                +^-------^--+    +---------^-+
                 |       |                 |
                 |       |                 |
                 |       |                 |
+----------------++   +--+--------------+  +-+-----------------+
|       (A)       |   |       (B)       |    |       (C)       |
|Component Variant|   |Component Variant|    |Component Variant|
|        +        |   |        +        |    |        +        |
|     Overlays    |   |     Overlays    |    |     Overlays    |
+-----------------+   +-----------------+    +-----------------+

           X n(variants) == off-the-shelf configuration

And a golang app with parameters of SERVICE_NAME=my-go-app and CONTEXT_PATH='/myApp'

+-----------------+
| Base Deployment |
+---------^-------+
          |
+---------+---------+              +------------+
| Golang Deployment |       +----->|Base Service|
+---------^---------+       |      +------------+
          |                 |
+---------+----------+      |   +-------------------+
| kustomization.yaml +--------->|Base VirtualService|
+--------------------+      |   +-------------------+
                            |
Resources a Deployment,     |      +------------+
Service, VirtualService,    +----->|Base Ingress|
and Ingress                        +------------+

[tkellen]

Thanks for sharing your workflow and thoughts @ricochet! I love the idea of a deprecation message as a resolution to #2071 (and mentioned as much over there). I have limited free time to work on this but I do have plans to share a set of configuration formats for a replacements transformer that could accomplish everything vars are presently being used for (at least, every workflow mentioned in this issue).

[benjamin-bergia]

Hi,
it's now one year that I have been managing our production and testing environments using exclusively kustomize. I also went through some of the patterns that have been posted, but have been able to refactor most of the cases to avoid variables.
Currently I have a single use case where I haven't been able to remove the variables and it is for the host field of my ingresses.

I have a dozen or so ingresses per environment, all of them using a subdomain but sharing the same domain. The only solution I have found have been to have a "global" configmap containing my domain name and a variable referencing this entry in the configmap.

I can then populate all my ingresses using things like
host: www.$(BASE_DOMAIN) or host: api.$(BASE_DOMAIN)
I also pass this configmap in the environment of my web servers when they need to know their own url.

All this is to be able to:

• use different domains on prod and testing without changing the subdomains
• adding new ingresses without copy/pasting domains or patches all over the place
• keep everything in sync across the whole environment

I also understand that this "global" configmap should be avoided but I have found that, unfortunately, we always end up with a handful of settings, mostly static, that have to be available environment wide and kept in sync. I find that keeping them in a separate configmap is need but unfortunately doesn't play well with kustomize configmap generators.

[candlerb]

Speaking as a newbie, I have one observation: kustomize's $(VAR) feature seems to overlap with kubernetes' own syntax to substitute environment variables in pod commands.

It's a potential point of confusion (for me anyway).

[mattmceuen]

Currently, the replacement transformer exists as a POC/ example plugin. My impression is that the next step is to reform it into a built-in plugin, is that correct? Are there any plans/timeline/critical path can be shared -- is it just a matter of time, or is there more evaluation that needs to happen first (or are we just waiting for someone to pick the work up)? Thanks!

[rdubya16]

@benjamin-bergia This is the exact same thing we are trying to solve. There still seems to be no way to avoid a small set of variables, the other ones we struggle with is things like the cluster name (required for tagging and alb-ingress), names of AZs cluster is deployed in (needed in CRDs for prometheus), and region. There is really no source of truth for these.

I still feel like kustomize is lacking the tools to solve this use case without resorting to some kind of meta-templating on top of kustomize. We are trying manage multiple clusters with only a small set of differing variables which results in copying a bunch of boilerplate or resorting to meta-templating.

[rehevkor5]

In the section, "I want to use diamonds":

top level overlay, called all, that merges
sibling levels dev, staging, prod (variants) by specifying
them as resources

and

With the above
kustomization file, one can deploy all environments with one
command

assume that the kustomizations being composed are variants and/or environments. However, that does not have to be the case in order to run into issues with different kustomizations declaring a variable with the same name, which breaks it ("var already encountered"). For example, after using kfctl to generate kustomizations, and using Kustomize itself to aggregate those and customize them for different environments, attempting to apply them with Kustomize fails due to the generated kustomizations declaring the same vars. The kustomizations are not environments or variants of the same thing: each one handles a different application or component of the overall system.

To me, as a total noobie to Kustomize (I prefer Helm's declarative style over Kustomize's imperative style), this is an issue of composability. If I have two kustomizations that work totally fine on their own, then I should be able to group both of them together into a 3rd kustomization, and the result should be identical either way. There shouldn't suddenly be leakages/side-effects happening between siblings.

[yanniszark]

Now that you have the powerful kyaml library, can't we do something like the following:

1. Replace all $(VAR) notation with setter comment notation: # {"$kpt-set":"VAR"}
2. Anchor the VAR to an object attribute, like what the current vars field does.
3. Namespace the var to the firstkustomization file that declares the var.

@Shell32-Natsu is there any discussion / plan for a timeline on an alternative? This seems to be one of the biggest pain-points of using kustomize today, as it breaks the encapsulation of the kustomization and prevents the user from composing bigger kustomizations out of smaller ones.

[monopole]

Enhancing vars will inexorably lead to kustomize becoming yet another crappy, half-baked domain specific configuration language.

@yanniszark you can use setters for many things, they're fine to use in kustomize.

Hoping someone will dust off the replacement transformer, or build a new one, so people can try it (#3492).

[wstrange]

Kustomize setters still show as "alpha", and there is confusion on the positioning of kustomize vs. kpt.

Some guidance on the formal direction would be very helpful.

[yanniszark]

@monopole this is great! I have to ask though, does this solution (replacement transformer) support partial substitution?
Partial substitution is one of the main uses of vars today. For example, to substitute a service name and namespace in a Certificate object: https://github.com/kubeflow/kfserving/blob/master/config/certmanager/certificate.yaml#L20

To elaborate some more, I believe there are two main use-cases that people use vars for today. @jlewi has very accurately described them in Kubeflow's kustomize best practices doc:
https://github.com/kubeflow/manifests/blob/master/docs/KustomizeBestPractices.md#eschew-vars

• Internal subsitution of fields Kustomize isn't aware of (e.g., https://github.com/kubeflow/kfserving/blob/master/config/certmanager/certificate.yaml#L20)
• Global configuration (e.g., cluster domain in all VirtualService objects).

The replacement transformer could probably solve the first one, but it needs to be more powerful and allow string substitutions. The second one is probably doable as well, if one uses a single global ConfigMap for the options and then refers to it for every replacement (but incurs a lot of repetition of multiple lines). Or a transformer for the specific use-case. Or setters, but it's not clear what kustomize's line is on their use.

[seh]

Adding on to @wstrange's comment above (#2052 (comment)), I asked about this confusing situation here: #3429 (comment).

[jlewi]

@yanniszark you can use setters for many things, they're fine to use in kustomize.

I believe the answer is yes. FWIW I use kpt+kustomize routinely. General pattern is

• kpt cfg set ${KUSTOMIZEDIR}
• kustomize build ${KUSTOMIZEDIR}

[GuyPaddock]

I really wish replacements had:

1. A string prefix and suffix feature.
2. The ability to reference the namespace that was set in the kustomization.yaml.

I am finding that I need to replace the namespace component of a nginx.ingress.kubernetes.io/auth-tls-secret annotation, since it's not getting updated automatically for a namespaced secret generated in each overlay.

I already have a ConfigMap that I am using as the source of values, and I have a namespace declaration in the kustomization.yaml file, but it appears that runs last (which kind of makes sense). So, I wound up having to manually declare the metadata.namespace within the ConfigMap for every overlay just so I could reference it.

Then, to replace the namespace part of the nginx.ingress.kubernetes.io/auth-tls-secret annotation, I ended up using the delimiter feature (with a delimiter of /) to target the zero-th component. That meant that in the base overlays I had to put a dummy namespace in that annotation.

[marshall007]

@GuyPaddock I believe what you are describing here is consistent with my proposal in #4080. It also addresses the even more painful use case of trying to replace namespaced values in container args.

[f3r73ch]

Don't want to be rude, but with the amount of issues raised and people complaining throughout all these years,... just enhance vars. It feels like a misperception of what the real world is like and that too much "dogmatism" makes this the typical example where we try to fit the foot into the shoe.

[Gikkman]

I've been spending a lot of time thinking about this myself, and while I agree with Kustomize's general idea that we want to avoid build-time side effects, I've come to think the real-world problem is unexpected or unknown side-effects. In several projects I've done now, I always seem to run into one or two places where I have to work around the lack of templating in kustomize. Either by doing kustomize build | envsubs | kubectl apply or by using a series of kustomize edit set, and in both of those cases, it feels like I am stepping away from having everything explicit in files.

In my opinion, the best way to handle variable replacement is similar to how Terraform does, where you explicitly declare what variables needs to be filled. Then you can either have a file to fill those variables (say in an overlay) or you can pull them from the environment. That way, you could even have a kustomize command to show you what variables are required to be able to render the manifest. You could also have a kustomize command to fill those variables if so wanted.

[pre]

@Gikkman summarized my thoughts exactly. We have written an envsubst tool of our own, reading substitute values from a file, for this exact purpose.

We wouldn’t need to do such a thing if there were variables/templating with kustomize.

[swapzero]

I recently had to resume a migration from vars to replacements and I though I should give it another try .
While generally replacements are ok, there is still the lack of replace capabilities inside text templates or directly in YAML like vars can do.

It turns out that it is possible, at least in the case of a JSON template which has a standard format and thanks to commas and double-quotes, you can do weird things like this:

https://github.com/swapzero/random/tree/main/kustomize

[mtrin]

@Gikkman nailed it.
We also had to build our own envsubst plugin.
Kubectl's server-side-apply dry-run also helps mitigate build time side effects. Which would be somewhat the equivalent of a terraform plan.