Allow helm values overrides to inject cluster labels via go templates #507

gregsidelinger · 2021-08-27T18:11:20Z

Signed-off-by: Greg Sidelinger gate@ilive4code.net

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

mattmattox · 2021-08-31T02:16:09Z

This would be really nice for when you want to expose a service on all clusters.
Example: longhorn.ClusterName.example.com

Currently, you can use variables like global.fleet.clusterLabels.management.cattle.io/cluster-display-name in your values but you can't append anything to the variable.

nickgerace · 2021-09-13T14:36:59Z

docs/gitrepo-structure.md

@@ -72,6 +72,10 @@ helm:
  # All labels on Rancher clusters are available using global.fleet.clusterLabels.LABELNAME
  # These can now be accessed directly as variables
    variableName: global.fleet.clusterLabels.LABELNAME
+  # They can also be access using go template logic


Small grammatical fixes:

In addition, they can be accessed using Go template logic

nickgerace · 2021-09-13T14:39:18Z

pkg/target/target_test.go

+	templateName, ok := bundle.Helm.Values.Data["templateName"]
+	if !ok {
+		t.Fatal("key templateName not found")
+	}
+
+	if templateName != "kubernetes.io/cluster/local" {
+		t.Fatal("unable to assert correct template")
+	}
+
+	templateMissing, ok := bundle.Helm.Values.Data["templateMissing"]
+	if !ok {
+		t.Fatal("key templateMissing not found")
+	}
+
+	if templateMissing != "{{ .missingValue }}" {
+		t.Fatal("unable to assert correct templateMising")
+	}
+
+	templateLogic, ok := bundle.Helm.Values.Data["templateLogic"]
+	if !ok {
+		t.Fatal("key templateLogic not found")
+	}
+
+	if templateLogic != "Non Prod" {
+		t.Fatal("unable to assert correct templateLogic")
+	}


nit: let's remove blank lines between the same template comparisons

nickgerace · 2021-09-13T14:40:51Z

pkg/target/target.go

@@ -520,6 +529,18 @@ func processLabelValues(valuesMap map[string]interface{}, clusterLabels map[stri
 			} else {
 				return fmt.Errorf("invalid_label_reference %s in key %s", valStr, key)
 			}
+		} else {
+			switch val.(type) {


We already gather valStr on line 523. Perhaps, we can refactor this code to use it instead. I'd prefer that we handle this scenario once.

I actually have an updated PR I was going to send in that changes how a lot of this stuff is done and does not have any extra string conversions in that area of the code.

After playing around with the configmap stuff in 0.3.6 I thought it would be fun to allow the value substitution to come from any overridden values like the ones imported from a configmap, not just the ones in helm.vlaues from the fleet.yaml. So I moved this code into the helm install function to do that. I also added basic cluster annotation support on top of the cluster one to deal with values that can not be stored in an label. It's a bigger PR but adds a lot more flexibility that has been preventing us from using fleet for more deployments.

gregsidelinger · 2021-09-13T15:23:55Z

#528 supersedes this PR

nickgerace · 2021-09-13T18:01:27Z

@gregsidelinger why are these PRs separate? I'd personally prefer one PR with discrete commits (or squashed, if small enough) in order to reduce context switching. However, it's probable that there's a perspective here that I am missing.

gregsidelinger · 2021-09-13T23:30:16Z

@nickgerace Because I'm not 100% sure if there are any major design issues with where/how I did things in #528. It was a much bigger change than this small PR which has more lines of testing than actual code. I would say look at #528 and if nothing major jumps out with the approach I will close this one and focus on it.

Cleanup extra string convirsion Cleanup extra whitespace in tests Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

nickgerace · 2021-09-15T15:55:44Z

Thanks @gregsidelinger for the clarification. I'll pause my review here and jump to the other one.

gregsidelinger · 2021-09-24T12:56:25Z

So after testing this branch and the one in #528 I think this is the PR that should be merged. I updated this PR with annotation support and added a few more test cases. As well as trapping errors from calling the go-templating that causes panics. Docs could get some for TLC as accessing labels and annotations with dashes in their name requires using index which I added a test case for. But I'm not sure if all of this should end up in the github-structure docs as that is starting to look full and out of place if there are too many examples of using go tempaltes.

The only real difference between this one and #528 is this supports only fleet.yaml values and valuesfiles. Configmaps and secrets in the destination cluster are not processed, which may have been overkill.

xMAC94x · 2021-11-17T19:33:04Z

Hi @gregsidelinger thanks for this PR, its exactly what i am looking for. I noticed there is a comment on your PR that need to be resolved before merging.
Fyi, i found out that even Yaml Anchors don't work in current release fleet version ( https://helm.sh/docs/chart_template_guide/yaml_techniques/#yaml-anchors )

aiyengar2

Thank you for your PR @gregsidelinger! Apologies for the late re-review, had some comments on this approach though.

aiyengar2 · 2022-02-09T21:30:55Z

docs/gitrepo-structure.md

+    variableName: "kubernetes.io/cluster/{{ .global.fleet.clusterLabels.LABELNAME }}"
+    variableName: "{{ if eq .global.fleet.clusterLabels.LABELNAME \"production\" }}Production Workload{{ else }}Non Prod{{ end }}"


What if the values provided here are intended to be Go templates that are parsed by the Helm chart? For example, values like this one found in kube-prometheus-stack that is expected to be a Go template supplied to the chart, as seen by the tpl call made here

Based on this approach, we would no longer be able to support Go template encoded strings provided via the values.yaml to be interpreted by Helm instead of Fleet.

That is an excellent point. I would think we could make this behavior opt in. So something like add parseValuesValues: true (that's a bad name fyi). Then if somebody wants to templatize something that has templates in it they would escape the template characters.

Yes, that is true. An early version of the code actually would leave any lines alone that returned an error from the go templating call allowing fleet to still handle values files with go templating strings in them. This however would not return errors to the user if they did something like this variableName: "kubernetes.io/cluster/{{ .global.fleet.clusterLabels.MISSING_LABEL }}" so I changed it as I forgot why I did it in the first place till you mentioned it.

Assuming I leave all clusterLabels prefixed with .global.fleet. a simple approach would be to only call to go templating if the string contained {{, }} and .global.fleet. which should in theory be safe. for most use cases. Or only support the templating on stuff in fleet.yaml and not values files.

I was thinking of helm.preProcessValues as the setting name to turn the templating on. I actually submitted a patch to @rajiteh for his PR #671 instead of more updates to my code. Granted I do have all the changes requested for this PR sitting locally but I think his approach is better you are should all review his PR.

aiyengar2 · 2022-02-09T21:36:26Z

pkg/target/target.go

+func processValues(valuesMap map[string]interface{}, clusterLabels map[string]string, clusterAnnotations map[string]string) (err error) {
+	labelPrefix := "global.fleet.clusterLabels."
+	annotationPrefix := "global.fleet.clusterAnnotations."
+	scopedClusterLables := map[string]interface{}{


Suggested change

scopedClusterLables := map[string]interface{}{

scopedClusterLabels := map[string]interface{}{

nit: incorrect spelling

aiyengar2 · 2022-02-09T22:00:26Z

pkg/target/target.go

+					return fmt.Errorf("invalid_label_reference %s in key %s", valStr, key)
+				}
+			} else if strings.HasPrefix(valStr, annotationPrefix) {
+				annoation := strings.TrimPrefix(valStr, annotationPrefix)


Suggested change

annoation := strings.TrimPrefix(valStr, annotationPrefix)

annotation := strings.TrimPrefix(valStr, annotationPrefix)

nit: typo

aiyengar2 · 2022-02-09T22:00:43Z

pkg/target/target.go

+				}
+			} else if strings.HasPrefix(valStr, annotationPrefix) {
+				annoation := strings.TrimPrefix(valStr, annotationPrefix)
+				annoationVal, annoationPresent := clusterLabels[annoation]


Suggested change

annoationVal, annoationPresent := clusterLabels[annoation]

annotationVal, annotationPresent := clusterLabels[annotation]

nit: typo

pkg/target/target.go

aiyengar2 · 2022-02-09T22:08:44Z

pkg/target/target.go

+				defer func() {
+					if r := recover(); r != nil {
+						err = fmt.Errorf("Failed to process template substitution for key '%s' with value '%s': [%v]", key, valStr, err)
+					}
+				}()


why do we need this? Is there a part of our code that panics?

yes, a bad go template can actually raise a panic.

is the panic introduced with this change or is it already in master and you just added the logging ?

aiyengar2 · 2022-02-09T22:11:31Z

pkg/target/target_test.go

+	bundle.Helm.Values.Data["error"] = "{{ .missingValue }}"
+	err = processValues(bundle.Helm.Values.Data, clusterLabels, clusterAnnotations)
+	if err == nil {
+		t.Fatalf("failed to error on {{ .missingValue }}")
+	}
+
+	bundle.Helm.Values.Data["error"] = "{{ .missingValue-panic }}"
+	err = processValues(bundle.Helm.Values.Data, clusterLabels, clusterAnnotations)
+	if err == nil {
+		t.Fatalf("failed to error on {{ .missingValue-panic }}")
+	}


How are these different tests?

One is designed to raise a panic from the go templating. I ran into a panic at one point in time so I wrote a test for it. If I recall correctly the - in the variable name actually causes the issue.

gregsidelinger · 2022-02-15T15:02:40Z

@xMAC94x @ibuildthecloud @aiyengar2
Design question. I spend some time over the weekend adding a boolean to turn on the templating to deal with charts that support the templating this would break and fixing the other issues pointed out. I also spend some time on PR #671 too and I like @rajiteh approach better. The flexibility of adding templateContext to the clusters seems better than relying upon labels and annotations to inject all data. And personally I like his format for accessing values inside the templating instead of the .global.fleet prefix I used.

	values := map[string]interface{}{
		"ClusterNamespace":   cluster.Namespace,
		"ClusterName":        cluster.Name,
		"ClusterLabels":      clusterLabels,
		"ClusterAnnotations": clusterAnnotations,
		"Values":             templateContext,
	}

So from a long term design perspective which approach is preferred?

xMAC94x · 2022-02-15T17:00:58Z

Agree, a special syntax is prob a better solution than put everything in labels, this would also get around of some restrictions as label values cannot contain all UTF-8 characters, right ?

prachidamle · 2022-03-18T00:12:09Z

@gregsidelinger Thank you for this PR and also the follow up design questions. I took a look at this PR and the conversations before where the problem of "how do we support templating for the charts that already have Go templates to be parsed by Helm engine". I see that you tried out the setting "helm.preProcessValues" method to turn on the Fleet templating when required.

So I think when the setting "helm.preProcessValues" is "true" it would indicate that Fleet needs to parse the templated values before passing down to Helm. Is that correct?

How would this make Fleet ignore processing of the templated charts intended to be processed by Helm engine? Are for such charts the setting should always be off?

I will go over the second PR #671 to understand the second approach and raise any questions I face.

rajiteh · 2022-03-23T20:20:45Z

@prachidamle author of PR #671 here. I have adopted the same config except inversed the config key. Essentially the config value and its default is helm.disablePreprocess: false, so the user could override it to true to disable templating. This will be particularly useful if they're sending gotpls as values which is the case for things like prometheus alertrules and etc.

However, a user could still utilize fleet-level helm templates and retain downstream templating ability by wrapping it in an escape block {{`{{ escaped_for_downstream }}`}}

prachidamle · 2022-03-24T16:47:00Z

@rajiteh so essentially when the gotpls are wrapped in an escape block {{{{ escaped_for_downstream }}}}, the fleet template parsing will unescape it for further parsing via helm?

rajiteh · 2022-03-24T17:05:28Z

Yes. The template string {{`{{ hello }}`}} will get templated out by gotpl (which is what the proposed solutions are utilizing) as {{ hello }}

https://go.dev/play/p/VvaLfCXnyhd

SheilaghM · 2022-03-28T19:14:35Z

@gregsidelinger @rajiteh - are we good to close this PR in favor of #671 and pursue that avenue?

gregsidelinger · 2022-03-28T19:50:38Z

@gregsidelinger @rajiteh - are we good to close this PR in favor of #671 and pursue that avenue?

I'm fine with focusing on #671. I did submit a PR to @rajiteh over the weekend to address the things brought up so far in the review so hopefully he is ok to merge those changes in.

SheilaghM · 2022-03-28T20:12:56Z

Thanks @gregsidelinger. I'm going to close this ticket then so we focus on #671.

Allow helm values overrides to inject cluster lables via go templates

c4ef182

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

gregsidelinger mentioned this pull request Aug 27, 2021

Global labels inside string #375

Closed

gregsidelinger added 4 commits August 27, 2021 15:06

Ran gofmt

a1414cc

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

Make the linter happy

838855f

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

Prefix go template labels with .global.fleet.clusterLabels

3f71e06

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

Ran gofmt, again

9df73ed

Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

leodotcloud requested a review from nickgerace September 13, 2021 13:32

nickgerace suggested changes Sep 13, 2021

View reviewed changes

Move if statements into existing switch statement

8cd86b1

Cleanup extra string convirsion Cleanup extra whitespace in tests Signed-off-by: Greg Sidelinger <gate@ilive4code.net>

nickgerace mentioned this pull request Sep 15, 2021

Enhanced value substitution #528

Closed

gregsidelinger added 2 commits September 21, 2021 15:49

Adding annotation support

4dd1fa1

Add new target tests

41c6b7f

leodotcloud requested review from nickgerace and StrongMonkey October 26, 2021 14:27

rajiteh mentioned this pull request Dec 4, 2021

Sprig Templating for Helm Values with Inputs from Cluster Resource #671

Merged

atsai1220 mentioned this pull request Dec 27, 2021

Ingress dynamic host name using fleet + kustomize #686

Open

SheilaghM requested review from prachidamle and aiyengar2 and removed request for StrongMonkey and nickgerace February 8, 2022 01:03

aiyengar2 requested changes Feb 9, 2022

View reviewed changes

SheilaghM closed this Mar 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow helm values overrides to inject cluster labels via go templates #507

Allow helm values overrides to inject cluster labels via go templates #507

gregsidelinger commented Aug 27, 2021

mattmattox commented Aug 31, 2021

nickgerace Sep 13, 2021

nickgerace Sep 13, 2021

nickgerace Sep 13, 2021

gregsidelinger Sep 13, 2021

gregsidelinger commented Sep 13, 2021

nickgerace commented Sep 13, 2021

gregsidelinger commented Sep 13, 2021

nickgerace commented Sep 15, 2021

gregsidelinger commented Sep 24, 2021

xMAC94x commented Nov 17, 2021

aiyengar2 left a comment

aiyengar2 Feb 9, 2022

ibuildthecloud Feb 10, 2022

gregsidelinger Feb 10, 2022

gregsidelinger Feb 17, 2022

aiyengar2 Feb 9, 2022

aiyengar2 Feb 9, 2022

aiyengar2 Feb 9, 2022

aiyengar2 Feb 9, 2022

gregsidelinger Feb 10, 2022

xMAC94x Feb 10, 2022

aiyengar2 Feb 9, 2022

gregsidelinger Feb 10, 2022

gregsidelinger commented Feb 15, 2022

xMAC94x commented Feb 15, 2022

prachidamle commented Mar 18, 2022 •

edited

rajiteh commented Mar 23, 2022

prachidamle commented Mar 24, 2022

rajiteh commented Mar 24, 2022 •

edited

SheilaghM commented Mar 28, 2022

gregsidelinger commented Mar 28, 2022

SheilaghM commented Mar 28, 2022

		variableName: "kubernetes.io/cluster/{{ .global.fleet.clusterLabels.LABELNAME }}"
		variableName: "{{ if eq .global.fleet.clusterLabels.LABELNAME \"production\" }}Production Workload{{ else }}Non Prod{{ end }}"

	scopedClusterLables := map[string]interface{}{
	scopedClusterLabels := map[string]interface{}{

	annoation := strings.TrimPrefix(valStr, annotationPrefix)
	annotation := strings.TrimPrefix(valStr, annotationPrefix)

	annoationVal, annoationPresent := clusterLabels[annoation]
	annotationVal, annotationPresent := clusterLabels[annotation]

Allow helm values overrides to inject cluster labels via go templates #507

Allow helm values overrides to inject cluster labels via go templates #507

Conversation

gregsidelinger commented Aug 27, 2021

mattmattox commented Aug 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsidelinger commented Sep 13, 2021

nickgerace commented Sep 13, 2021

gregsidelinger commented Sep 13, 2021

nickgerace commented Sep 15, 2021

gregsidelinger commented Sep 24, 2021

xMAC94x commented Nov 17, 2021

aiyengar2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsidelinger commented Feb 15, 2022

xMAC94x commented Feb 15, 2022

prachidamle commented Mar 18, 2022 • edited

rajiteh commented Mar 23, 2022

prachidamle commented Mar 24, 2022

rajiteh commented Mar 24, 2022 • edited

SheilaghM commented Mar 28, 2022

gregsidelinger commented Mar 28, 2022

SheilaghM commented Mar 28, 2022

prachidamle commented Mar 18, 2022 •

edited

rajiteh commented Mar 24, 2022 •

edited