Helm Chart 4.3.0+ breaks secrets

[296951]

Describe the bug

I currently use Helm Chart Version 4.2.21 and all works beautifully. If I change to 4.3.0+ My secrets break.

With the 4.2.21 Helm Chart My Jenkins-0 container has and env var SECRETS=[filepath]. [filepath] is a location where various secrets are mounted.

With 4.3.0+ Helm Chart my Jenkins-0 container is missing this env var. The odd thing is if I run a helm template I seen the correct env var in my templated helm output and it is correct. The secrets are even mounted correctly to the location in the container, however, the env var is missing.

I am using SAML and have controller.adminSecret: false and I am specifying my own mounts and a custom mountpoint for secrets.

Version of Helm and Kubernetes

- Helm: version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

- Kubernetes: Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:21:44Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.14-eks-c12679a", GitCommit:"05d192f0de17608d98e17761ad3cffa9a6407f2f", GitTreeState:"clean", BuildDate:"2023-05-22T23:41:27Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}

Chart version

4.3.x

What happened?

1. I deploy jenkins using values files that work with Helm Chart version 4.2.21 with helm chart version 4.3.x.
2. Local Jenkins provider secrets mounted to a filepath from kubernetes secrets appear empty in the Jenkins UI
3. The `SECRETS=[filepath]` env var is missing in the 4.3.x helm chart deployed container.
   - The env var is in templated helm output by a `helm template` command using these values files.

What you expected to happen?

1. I deploy jenkins using values files that work with Helm Chart version 4.2.21 with helm chart version 4.3.x.
2. Local Jenkins provider secrets mounted to a filepath from kubernetes secrets are populated with required data in the Jenkins UI and work when I use them.
3. The SECRETS=[filepath] env var should exist 4.3.x helm chart deployed container.

How to reproduce it

My Jenkins deployment is very complex and I am not sure how to distill this down to something simple.

Anything else we need to know?

The ONLY Difference in the templated Helm output between 4.2.21 and 4.3.0 is the Helm chart version and the placement of the following block under template.spec.containers.env::

            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: JAVA_OPTS
              value: >-
                 -Dcasc.reload.token=$(POD_NAME) 
            - name: JENKINS_OPTS
              value: >-
                --prefix=/jenkins  --webroot=/var/jenkins_cache/war 
            - name: JENKINS_SLAVE_AGENT_PORT
              value: "50000"

[296951]

This might be a Kubernetes or a Helm issue. We have now seen this on the 4.2.21 Jenkins helm chart.

SOMETIMES the Jenkins statefulset has the SECRETS env variable, sometimes it does not. Redeploying via helm with no changes to the values files seems to fix it so does editing the statefulset with kubectl and adding the SECRETS env variable.

[296951]

I endend up writing a test that I run after I deploy jenkins that looks like:

 kubectl get statefulsets.apps jenkins -o json |jq -r '.spec.template.spec.containers[] | select(.name | contains("jenkins")).env[] | select(.name=="SECRETS").value'

If this returns nothing I sleep 5 seconds and rerun the deploy using the exact same helm command and values files.

This resolves my issue.

Leaving this open just in case someone wants to look at it.

Close it if you wish.